Re: Custom logging function

[Peter Otten]

zljubi...@gmail.com wrote:

>> You create two stream handlers that both log to stderr -- one with
>> basicConfig() and one explicitly in your code. That's probably not what
>> you want.
> 
> How can I just add terminator = '\r\n' to the code bellow?
> Where should I put it?

I already answered that in my first post in the thread.
Spoon-feeding below ;)

> import logging
> 
> LOG_LEVEL = logging.getLevelName('DEBUG')
> 
> logging.basicConfig(level=LOG_LEVEL,
> format='%(asctime)s %(levelname)s %(name)s
> %(funcName)-20s %(message)s', datefmt='%d.%m.%Y
> %H:%M:%S')

[handler] = logging.getLogger().handlers
handler.terminator = "\r\n"

> logging.info('Main script started')


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Elegant hack or gross hack? TextWrapper and escape codes

[Peter Otten]

Chris Angelico wrote:

> Situation: A terminal application. Requirement: Display nicely-wrapped
> text. With colour codes in it. And that text might be indented to any
> depth.
> 
> label = f"{indent}\U0010cc32{code}\U0010cc00
> @{tweet['user']['screen_name']}: " wrapper = textwrap.TextWrapper(
> initial_indent=label,
> subsequent_indent=indent + " " * 12,
> width=shutil.get_terminal_size().columns,
> break_long_words=False, break_on_hyphens=False, # Stop URLs from
> breaking
> )
> for line in tweet["full_text"].splitlines():
> print(wrapper.fill(line)
> .replace("\U0010cc32", "\x1b[32m\u2026")
> .replace("\U0010cc00", "\u2026\x1b[0m")
> )
> wrapper.initial_indent = wrapper.subsequent_indent # For
> subsequent lines, just indent them
> 
> 
> The parameter "indent" is always some number of spaces (possibly
> zero). If I simply include the escape codes in the label, their
> characters will be counted, and the first line will be shorter. Rather
> than mess with how textwrap defines text, I just replace the escape
> codes *and one other character* with a placeholder. In the final
> display, \U0010cc32 means "colour code 32 and an ellipsis", and
> \U0010cc00 means "colour code 0 and an ellipsis", so textwrap
> correctly counts them as one character each.
> 
> So what do you folks think? Is this a gloriously elegant way to
> collapse nonprinting text, or is it a gross hacky mess 

Yes ;)

> that's going to cause problems?

Probably not. 

I had a quick look at the TextWrapper class, and it doesn't really lend 
itself to clean and elegant customisation. However, my first idea to 
approach this problem was to patch the len() builtin:

import re
import textwrap

text = """The parameter "indent" is always some number of spaces (possibly
zero). If I simply include the escape codes in the label, their
characters will be counted, and the first line will be shorter. Rather
than mess with how textwrap defines text, I just replace the escape
codes *and one other character* with a placeholder. In the final
display, \U0010cc32 means "colour code 32 and an ellipsis", and
\U0010cc00 means "colour code 0 and an ellipsis", so textwrap
correctly counts them as one character each.
"""

print(textwrap.fill(text, width=40))

# add some color to the text sample
GREEN =  "\x1b[32m"
NORMAL = "\x1b[0m"
parts = text.split(" ")
parts[::2] = [GREEN + p + NORMAL for p in parts[::2]]
ctext = " ".join(parts)

# wrong wrapping
print(textwrap.fill(ctext, width=40))

# fixed wrapping
def color_len(s):
return len(re.compile("\x1b\[\d+m").sub("", s))

textwrap.len = color_len
print(textwrap.fill(ctext, width=40))

The output of my ad-hoc test script looks OK. However, I did not try to 
understand the word-breaking regexes, so I don't know if the escape codes 
can be spread across words which would confuse color_len(). Likewise, I have 
no idea if textwrap can cope with zero-length chunks.

But at least now you have two -- elegant or gross -- hacks to choose from ;)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Elegant hack or gross hack? TextWrapper and escape codes

[Chris Angelico]

On Thu, May 28, 2020 at 5:54 PM Peter Otten <__pete...@web.de> wrote:
>
> Chris Angelico wrote:
>
> > Situation: A terminal application. Requirement: Display nicely-wrapped
> > text. With colour codes in it. And that text might be indented to any
> > depth.
> >
> > label = f"{indent}\U0010cc32{code}\U0010cc00
> > @{tweet['user']['screen_name']}: " wrapper = textwrap.TextWrapper(
> > initial_indent=label,
> > subsequent_indent=indent + " " * 12,
> > width=shutil.get_terminal_size().columns,
> > break_long_words=False, break_on_hyphens=False, # Stop URLs from
> > breaking
> > )
> > for line in tweet["full_text"].splitlines():
> > print(wrapper.fill(line)
> > .replace("\U0010cc32", "\x1b[32m\u2026")
> > .replace("\U0010cc00", "\u2026\x1b[0m")
> > )
> > wrapper.initial_indent = wrapper.subsequent_indent # For
> > subsequent lines, just indent them
> >
> >
> > The parameter "indent" is always some number of spaces (possibly
> > zero). If I simply include the escape codes in the label, their
> > characters will be counted, and the first line will be shorter. Rather
> > than mess with how textwrap defines text, I just replace the escape
> > codes *and one other character* with a placeholder. In the final
> > display, \U0010cc32 means "colour code 32 and an ellipsis", and
> > \U0010cc00 means "colour code 0 and an ellipsis", so textwrap
> > correctly counts them as one character each.
> >
> > So what do you folks think? Is this a gloriously elegant way to
> > collapse nonprinting text, or is it a gross hacky mess
>
> Yes ;)

... I should have expected a "yes" response to an either-or question.
Silly of me. :)

> > that's going to cause problems?
>
> Probably not.
>
> I had a quick look at the TextWrapper class, and it doesn't really lend
> itself to clean and elegant customisation. However, my first idea to
> approach this problem was to patch the len() builtin:
>
> import re
> import textwrap
>
> text = """The parameter "indent" is always some number of spaces (possibly
> zero). If I simply include the escape codes in the label, their
> characters will be counted, and the first line will be shorter. Rather
> than mess with how textwrap defines text, I just replace the escape
> codes *and one other character* with a placeholder. In the final
> display, \U0010cc32 means "colour code 32 and an ellipsis", and
> \U0010cc00 means "colour code 0 and an ellipsis", so textwrap
> correctly counts them as one character each.
> """
>
> print(textwrap.fill(text, width=40))
>
> # add some color to the text sample
> GREEN =  "\x1b[32m"
> NORMAL = "\x1b[0m"
> parts = text.split(" ")
> parts[::2] = [GREEN + p + NORMAL for p in parts[::2]]
> ctext = " ".join(parts)
>
> # wrong wrapping
> print(textwrap.fill(ctext, width=40))
>
> # fixed wrapping
> def color_len(s):
> return len(re.compile("\x1b\[\d+m").sub("", s))
>
> textwrap.len = color_len
> print(textwrap.fill(ctext, width=40))
>
> The output of my ad-hoc test script looks OK. However, I did not try to
> understand the word-breaking regexes, so I don't know if the escape codes
> can be spread across words which would confuse color_len(). Likewise, I have
> no idea if textwrap can cope with zero-length chunks.
>
> But at least now you have two -- elegant or gross -- hacks to choose from ;)
>

Yeah, I thought of this originally as a challenge in redefining the
concept of "length". But the trouble is that it might not always be
the len() function that figures out the length - there might be a
regex with a size threshold or any number of other things that
effectively think about the length of the string.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Elegant hack or gross hack? TextWrapper and escape codes

[Peter Otten]

Chris Angelico wrote:

> On Thu, May 28, 2020 at 5:54 PM Peter Otten <__pete...@web.de> wrote:

>> But at least now you have two -- elegant or gross -- hacks to choose from
>> ;)
>>
> 
> Yeah, I thought of this originally as a challenge in redefining the
> concept of "length". But the trouble is that it might not always be
> the len() function that figures out the length - there might be a
> regex with a size threshold or any number of other things that
> effectively think about the length of the string.

So start with your hack, and if the pain becomes too hard do it right, 
perhaps making TextWrapper a bit more intelligible in the process.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system

[Rahul Gupta]

On Thursday, May 28, 2020 at 11:20:05 AM UTC+5:30, Rahul Gupta wrote:
> I am having a Ubuntu system which has 125 Gb of RAM. I executed few python 
> scripts on that system. Those scripts uses numpy arrays and pandas. Now 
> execution was over but still 50 gb of RAM and 2 Gb cache and 8.4 Gb of swap 
> is occupied. At this moment nothing is running on the system. I have googled 
> it. Most of th result shows that python garbage collector is poor in 
> performance. I want this memory to be cleaned and re claim. One of the 
> easiest way is to restart the system but i dont want to restart i want a way 
> to do this when the system is up and running. Kindly tell me how to do this. 
> Thanks
Yes i am sure 125 gb of ram is there.
And you talked about refrences 
see these links
https://stackoverflow.com/questions/39100971/how-do-i-release-memory-used-by-a-pandas-dataframe
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system

[Chris Angelico]

On Thu, May 28, 2020 at 6:26 PM Rahul Gupta  wrote:
>
> On Thursday, May 28, 2020 at 11:20:05 AM UTC+5:30, Rahul Gupta wrote:
> > I am having a Ubuntu system which has 125 Gb of RAM. I executed few python 
> > scripts on that system. Those scripts uses numpy arrays and pandas. Now 
> > execution was over but still 50 gb of RAM and 2 Gb cache and 8.4 Gb of swap 
> > is occupied. At this moment nothing is running on the system. I have 
> > googled it. Most of th result shows that python garbage collector is poor 
> > in performance. I want this memory to be cleaned and re claim. One of the 
> > easiest way is to restart the system but i dont want to restart i want a 
> > way to do this when the system is up and running. Kindly tell me how to do 
> > this. Thanks
> Yes i am sure 125 gb of ram is there.
> And you talked about refrences
> see these links
> https://stackoverflow.com/questions/39100971/how-do-i-release-memory-used-by-a-pandas-dataframe
> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm
>

Both of those are extremely old. Also, you said that you had
terminated the Python process, so neither is relevant.

You need to look elsewhere for your memory usage. If Python isn't
running, it's not Python that's using up your memory.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[BlindAnagram]

On 27/05/2020 23:39, Roel Schroeven wrote:
> BlindAnagram schreef op 27/05/2020 om 22:55:
>> On 27/05/2020 18:42, Roel Schroeven wrote:
>>> BlindAnagram schreef op 27/05/2020 om 18:53:
 Its not my bug to fix - the semantics of what I send is very clear on
 any Windows system.
>>>
>>> That's the first time I see any mention of those semantics, and I've
>>> been using Windows since the Windows 3.1 days (and MS-DOS before that,
>>> since 3.2 IIRC).
>>
>> Well I can only say that I am surpised that anyone working on Windows
>> code development for as long as you have doesn't know that a path that
>> ends in '\\' is a path to a directory.
> 
> A path ending in a backslash cannot refer to a file, that much is true.
> So if you have such a path and you insist that path is not an incomplete
> path, than it must out of necessity be a directory. But a path not
> ending in a backslash can refer to a directory just as well.

I understand that directory paths do not necessarily end in a '\\'.

But I am NOT claiming that paths on Windows are paths to directories if
and only if they end in '\\' - there is only ONE 'if' in my claim.

And this can be refuted with a single counterexample.

> If those semantics are as clear as you say, it shouldn't be too
> difficult to point us to a section in the Windows documentation that
> confirms that.

I don't know whether, in the many gigabytes of MS documentation that
exists since Windows first emerged, this has ever been explicitly
stated.  And I am not going to look as it isn't important to me that
people accept my claim.  But I do find it interesting that it seems
important to others here that I should withdraw it.

[snip]

> I find no hints of adding a backslash at the end to indicate directories.
> If you can point me to convincing evidence in the documentation I'll
> change my mind.

And if you find a counterexample, I will change mine.

> I'm sorry if my language seems abrasive; that is not my intention. I'm
> only trying to help you and clear up a misunderstanding (which could be
> mine, if it turns out you're right after all).

No need to apologise, I don't have any problem with your input to this
exchange.
-- 
https://mail.python.org/mailman/listinfo/python-list

Constructing mime image attachment

[Joseph L. Casale]

I have some json encoded input for nodemailer 
(https://nodemailer.com/message/embedded-images)
where the path key is a string value which contains the base64 encoded data 
such as:

{
html: 'Embedded image: ',
attachments: [{
filename: 'image.png',
path: 'data:image/png;base64,iVBORw',
cid: 'uni...@nodemailer.com'
}]
}

Does an approach exist to construct a MIMEImage object with the existing
base64 string without parsing it and decoding it?

Thanks,
jlc
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Eryk Sun]

On 5/26/20, BlindAnagram  wrote:
>
> But if I try to make the directory myself (as I tried first):
>
>   join(base, '..\\..\\', 'build', '\\')
>
> I obtain:
>
> 'C:\\'
>
> The documentation says that an absolute path in the parameter list for
> join will discard all previous parameters but '\\' is not an absoute path!

First, to be clear, a rooted path in Windows is not absolute (*), but
that's not relevant here. What happens is that joining a path is like
issuing successive "cd" commands in the shell, and "cd \" takes you to
the root path of the current drive. The implementation of
ntpath.join() similarly tracks the current drive via
ntpath.splitdrive() while joining components.

---

(*)
When opened, a rooted path gets resolved relative to the drive of the
process working directory. Except in a relative symlink, where a
rooted target path is relative to the device of the parsed path of the
symlink. For example, if the parsed, native NT path of a symlink is
r"\Device\HarddiskVolume10\path\to\symlink", and the symlink targets
r"\path\to\file", then it gets resolved as
r"\Device\HarddiskVolume10\path\to\file".

ntpath.isabs incorrectly classifies rooted paths as absolute, and its
use is actually breaking part of the fallback code in the new
implementation of ntpath.realpath when manually resolving relative
symlinks. I've known of the problem for a few years and have discussed
it one one or two issues on the tracker, but correcting the behavior
of such a commonly used function is not an easy change to propose.
Usually mistakes made early on in development become permanent warts.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Eryk Sun]

On 5/27/20, Chris Angelico  wrote:
> On Thu, May 28, 2020 at 7:07 AM BlindAnagram 
> wrote:
>> You can define a path however you want but it won't change the fact that
>> on Windows a path that ends in '\\' is inherently a path to a directory.
>
> Citation needed.

See [MS-FSA] 2.1.5.1 Server Requests an Open of a File [1]. Here are
the relevant statements:

The operation MUST be failed with STATUS_OBJECT_NAME_INVALID under
any of the following conditions:

  * If PathName contains a trailing backslash and
CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE.


If PathName contains a trailing backslash:
  * If StreamTypeToOpen is DataStream or
CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE, the operation MUST
be failed with STATUS_OBJECT_NAME_INVALID

Internally, WinAPI CreateFileW calls NTAPI NtCreateFile with the
create option FILE_NON_DIRECTORY_FILE (i.e. only open or create a data
file stream), unless backup semantics are requested in order to be
able to open a directory (i.e. an index stream), in which case the
call uses neither FILE_NON_DIRECTORY_FILE nor FILE_DIRECTORY_FILE and
leaves it up to the path name. In other words, with backup semantics,
if you need to ensure that only a directory is opened, add a trailing
slash.

NTAPI STATUS_OBJECT_NAME_INVALID translates to WinAPI
ERROR_INVALID_NAME (123), which translates to C EINVAL (22).

[1]: 
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[BlindAnagram]

On 28/05/2020 14:51, Eryk Sun wrote:
> On 5/27/20, Chris Angelico  wrote:
>> On Thu, May 28, 2020 at 7:07 AM BlindAnagram 
>> wrote:
>>> You can define a path however you want but it won't change the fact that
>>> on Windows a path that ends in '\\' is inherently a path to a directory.
>>
>> Citation needed.
> 
> See [MS-FSA] 2.1.5.1 Server Requests an Open of a File [1]. Here are
> the relevant statements:
> 
> The operation MUST be failed with STATUS_OBJECT_NAME_INVALID under
> any of the following conditions:
> 
>   * If PathName contains a trailing backslash and
> CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE.
> 
> 
> If PathName contains a trailing backslash:
>   * If StreamTypeToOpen is DataStream or
> CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE, the operation MUST
> be failed with STATUS_OBJECT_NAME_INVALID
> 
> Internally, WinAPI CreateFileW calls NTAPI NtCreateFile with the
> create option FILE_NON_DIRECTORY_FILE (i.e. only open or create a data
> file stream), unless backup semantics are requested in order to be
> able to open a directory (i.e. an index stream), in which case the
> call uses neither FILE_NON_DIRECTORY_FILE nor FILE_DIRECTORY_FILE and
> leaves it up to the path name. In other words, with backup semantics,
> if you need to ensure that only a directory is opened, add a trailing
> slash.
> 
> NTAPI STATUS_OBJECT_NAME_INVALID translates to WinAPI
> ERROR_INVALID_NAME (123), which translates to C EINVAL (22).
> 
> [1]: 
> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40
> 
Thank you for making the effort to answer a number of issues raaised in
this thread.  I much appreciate your input.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Eryk Sun]

On 5/28/20, BlindAnagram  wrote:
>
> Thank you for making the effort to answer a number of issues raaised in
> this thread.  I much appreciate your input.

For a more practical POV, see the topic on "File System Navigation"
[1] for the C++ standard  API. In the C++ standard
library, trailing slashes are not stripped and path
p0(L"C:/Documents") compares as less than path p1(L"C:/Documents/").

Also, regarding whether rooted paths without a drive are absolute,
you'll find that C++ is_absolute() requires has_root_name() &&
has_root_directory() in Windows. A rooted path without a drive is thus
not absolute in the C++ standard library.

[1]: 
https://docs.microsoft.com/en-us/cpp/standard-library/file-system-navigation?view=vs-2019
https://docs.microsoft.com/en-us/cpp/standard-library/file-system-navigation?view=vs-2019
[2]: 
https://docs.microsoft.com/en-us/cpp/standard-library/path-class?view=vs-2019#is_absolute
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[BlindAnagram]

On 27/05/2020 23:39, Roel Schroeven wrote:

> I find no hints of adding a backslash at the end to indicate directories.
> 
> If you can point me to convincing evidence in the documentation I'll
> change my mind.

See the post from Eryk Sun, later in this thread.
-- 
https://mail.python.org/mailman/listinfo/python-list

SQLAlchemy & Postgresql

[Buddy Peacock]

Hello group,
I have a pretty good background in MySQL & MSSQL as well as VB & Php, but I
am new to Python and Postgresql.

I am taking a class and working on a project to insert authors and books
into a table.  My code for this is:
===
import csv
import os

from flask import Flask
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker

app = Flask(__name__)

# Configure session to use filesystem
app.config["SESSION_PERMANENT"] = False
app.config["SESSION_TYPE"] = "filesystem"
# Session(app)

# Set up database
engine = create_engine(os.getenv("DATABASE_URL"))
db = scoped_session(sessionmaker(bind=engine))

print (os.getenv("DATABASE_URL"))

def main():
f = open("books.csv")
reader = csv.reader(f)
for isbn, title, author, year in reader:
if db.execute("SELECT * FROM authors WHERE name = :author",
{"name": author}).rowcount == 0:
db.execute("INSERT INTO authors (name) VALUES (author)")

print(f" {author} was read.")
db.commit()

if __name__ == "__main__":
main()

If I comment out the "if" statement then the authors all print on screen.
But when I run this with the if statement I am getting a message that says:
ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
value is required for bind parameter 'author'
[SQL: SELECT * FROM authors WHERE name = %(author)s]
[parameters: [{'name': 'author'}]]

What am I missing?


Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
(920) 740-3411
linkedin.com/in/buddypeacock 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Abdur-Rahmaan Janhangeer]

Greetings,


where did you define your models?

Kind Regards,


Abdur-Rahmaan Janhangeer

https://www.github.com/Abdur-RahmaanJ

Mauritius

sent from gmail client on Android, that's why the signature is so ugly.

On Thu, 28 May 2020, 22:22 Buddy Peacock,  wrote:

> Hello group,
> I have a pretty good background in MySQL & MSSQL as well as VB & Php, but I
> am new to Python and Postgresql.
>
> I am taking a class and working on a project to insert authors and books
> into a table.  My code for this is:
> ===
> import csv
> import os
>
> from flask import Flask
> from sqlalchemy import create_engine
> from sqlalchemy.orm import scoped_session, sessionmaker
>
> app = Flask(__name__)
>
> # Configure session to use filesystem
> app.config["SESSION_PERMANENT"] = False
> app.config["SESSION_TYPE"] = "filesystem"
> # Session(app)
>
> # Set up database
> engine = create_engine(os.getenv("DATABASE_URL"))
> db = scoped_session(sessionmaker(bind=engine))
>
> print (os.getenv("DATABASE_URL"))
>
> def main():
> f = open("books.csv")
> reader = csv.reader(f)
> for isbn, title, author, year in reader:
> if db.execute("SELECT * FROM authors WHERE name = :author",
> {"name": author}).rowcount == 0:
> db.execute("INSERT INTO authors (name) VALUES (author)")
>
> print(f" {author} was read.")
> db.commit()
>
> if __name__ == "__main__":
> main()
> 
> If I comment out the "if" statement then the authors all print on screen.
> But when I run this with the if statement I am getting a message that says:
> ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
> value is required for bind parameter 'author'
> [SQL: SELECT * FROM authors WHERE name = %(author)s]
> [parameters: [{'name': 'author'}]]
>
> What am I missing?
>
>
> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
> (920) 740-3411
> linkedin.com/in/buddypeacock 
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Buddy Peacock]

I'm not sure what you mean by models?  Am I missing something in my
environment perhaps?

Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
(920) 740-3411
linkedin.com/in/buddypeacock 




On Thu, May 28, 2020 at 2:26 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> Greetings,
>
>
> where did you define your models?
>
> Kind Regards,
>
>
> Abdur-Rahmaan Janhangeer
>
> https://www.github.com/Abdur-RahmaanJ
>
> Mauritius
>
> sent from gmail client on Android, that's why the signature is so ugly.
>
> On Thu, 28 May 2020, 22:22 Buddy Peacock,  wrote:
>
>> Hello group,
>> I have a pretty good background in MySQL & MSSQL as well as VB & Php, but
>> I
>> am new to Python and Postgresql.
>>
>> I am taking a class and working on a project to insert authors and books
>> into a table.  My code for this is:
>> ===
>> import csv
>> import os
>>
>> from flask import Flask
>> from sqlalchemy import create_engine
>> from sqlalchemy.orm import scoped_session, sessionmaker
>>
>> app = Flask(__name__)
>>
>> # Configure session to use filesystem
>> app.config["SESSION_PERMANENT"] = False
>> app.config["SESSION_TYPE"] = "filesystem"
>> # Session(app)
>>
>> # Set up database
>> engine = create_engine(os.getenv("DATABASE_URL"))
>> db = scoped_session(sessionmaker(bind=engine))
>>
>> print (os.getenv("DATABASE_URL"))
>>
>> def main():
>> f = open("books.csv")
>> reader = csv.reader(f)
>> for isbn, title, author, year in reader:
>> if db.execute("SELECT * FROM authors WHERE name = :author",
>> {"name": author}).rowcount == 0:
>> db.execute("INSERT INTO authors (name) VALUES (author)")
>>
>> print(f" {author} was read.")
>> db.commit()
>>
>> if __name__ == "__main__":
>> main()
>> 
>> If I comment out the "if" statement then the authors all print on screen.
>> But when I run this with the if statement I am getting a message that
>> says:
>> ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
>> value is required for bind parameter 'author'
>> [SQL: SELECT * FROM authors WHERE name = %(author)s]
>> [parameters: [{'name': 'author'}]]
>>
>> What am I missing?
>>
>>
>> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
>> (920) 740-3411
>> linkedin.com/in/buddypeacock 
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Abdur-Rahmaan Janhangeer]

Well,


See

for isbn, title, author, year in reader:
if db.execute("SELECT * FROM authors WHERE name = :author",
{"name": author}).rowcount == 0:
db.execute("INSERT INTO authors (name) VALUES (author)")

but i don't see any code for creating the authors table

also did you mean "name = author" instead of "name = :author"

3rd thing is that since you are using SQLALchemy, you should not be using
pure SQL

What i mean by models is something like this:
https://github.com/Abdur-rahmaanJ/shopyo/blob/dev/shopyo/modules/product/models.py

where you define code and your table is created the link uses
flask-sqlalchemy but pure sqlalchemy should
be somewhat similar

4th thing: when using

for isbn, title, author, year in reader:

without flask, do you actually see a value for author?

Kind Regards,

Abdur-Rahmaan Janhangeer
compileralchemy  | blog

github 
Mauritius


On Thu, May 28, 2020 at 10:31 PM Buddy Peacock 
wrote:

> I'm not sure what you mean by models?  Am I missing something in my
> environment perhaps?
>
> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
> (920) 740-3411
> linkedin.com/in/buddypeacock 
>
>
>
>
> On Thu, May 28, 2020 at 2:26 PM Abdur-Rahmaan Janhangeer <
> arj.pyt...@gmail.com> wrote:
>
>> Greetings,
>>
>>
>> where did you define your models?
>>
>> Kind Regards,
>>
>>
>> Abdur-Rahmaan Janhangeer
>>
>> https://www.github.com/Abdur-RahmaanJ
>>
>> Mauritius
>>
>> sent from gmail client on Android, that's why the signature is so ugly.
>>
>> On Thu, 28 May 2020, 22:22 Buddy Peacock, 
>> wrote:
>>
>>> Hello group,
>>> I have a pretty good background in MySQL & MSSQL as well as VB & Php,
>>> but I
>>> am new to Python and Postgresql.
>>>
>>> I am taking a class and working on a project to insert authors and books
>>> into a table.  My code for this is:
>>> ===
>>> import csv
>>> import os
>>>
>>> from flask import Flask
>>> from sqlalchemy import create_engine
>>> from sqlalchemy.orm import scoped_session, sessionmaker
>>>
>>> app = Flask(__name__)
>>>
>>> # Configure session to use filesystem
>>> app.config["SESSION_PERMANENT"] = False
>>> app.config["SESSION_TYPE"] = "filesystem"
>>> # Session(app)
>>>
>>> # Set up database
>>> engine = create_engine(os.getenv("DATABASE_URL"))
>>> db = scoped_session(sessionmaker(bind=engine))
>>>
>>> print (os.getenv("DATABASE_URL"))
>>>
>>> def main():
>>> f = open("books.csv")
>>> reader = csv.reader(f)
>>> for isbn, title, author, year in reader:
>>> if db.execute("SELECT * FROM authors WHERE name = :author",
>>> {"name": author}).rowcount == 0:
>>> db.execute("INSERT INTO authors (name) VALUES (author)")
>>>
>>> print(f" {author} was read.")
>>> db.commit()
>>>
>>> if __name__ == "__main__":
>>> main()
>>> 
>>> If I comment out the "if" statement then the authors all print on screen.
>>> But when I run this with the if statement I am getting a message that
>>> says:
>>> ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
>>> value is required for bind parameter 'author'
>>> [SQL: SELECT * FROM authors WHERE name = %(author)s]
>>> [parameters: [{'name': 'author'}]]
>>>
>>> What am I missing?
>>>
>>>
>>> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
>>> (920) 740-3411
>>> linkedin.com/in/buddypeacock 
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>>>
>>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Peter Otten]

Buddy Peacock wrote:

> Hello group,
> I have a pretty good background in MySQL & MSSQL as well as VB & Php, but
> I am new to Python and Postgresql.
> 
> I am taking a class and working on a project to insert authors and books
> into a table.  My code for this is:
> ===
> import csv
> import os
> 
> from flask import Flask
> from sqlalchemy import create_engine
> from sqlalchemy.orm import scoped_session, sessionmaker
> 
> app = Flask(__name__)
> 
> # Configure session to use filesystem
> app.config["SESSION_PERMANENT"] = False
> app.config["SESSION_TYPE"] = "filesystem"
> # Session(app)
> 
> # Set up database
> engine = create_engine(os.getenv("DATABASE_URL"))
> db = scoped_session(sessionmaker(bind=engine))
> 
> print (os.getenv("DATABASE_URL"))
> 
> def main():
> f = open("books.csv")
> reader = csv.reader(f)
> for isbn, title, author, year in reader:
> if db.execute("SELECT * FROM authors WHERE name = :author",
> {"name": author}).rowcount == 0:
> db.execute("INSERT INTO authors (name) VALUES (author)")
> 
> print(f" {author} was read.")
> db.commit()
> 
> if __name__ == "__main__":
> main()
> 
> If I comment out the "if" statement then the authors all print on screen.
> But when I run this with the if statement I am getting a message that
> says: ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError)
> A value is required for bind parameter 'author'
> [SQL: SELECT * FROM authors WHERE name = %(author)s]
> [parameters: [{'name': 'author'}]]
> 
> What am I missing?

The variable name for name is :author. Therefore I think you have to use
"author" as the key in the dict:

db.execute("SELECT * FROM authors WHERE name = :author", {"author": author})


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Buddy Peacock]

Thanks for your response Abdur-Rahmaan,

The tables already exist and I have tried both  "name = author" and "name =
:author"

Regards,

Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
(920) 740-3411
linkedin.com/in/buddypeacock 




On Thu, May 28, 2020 at 2:44 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> Well,
>
>
> See
>
> for isbn, title, author, year in reader:
> if db.execute("SELECT * FROM authors WHERE name = :author",
> {"name": author}).rowcount == 0:
> db.execute("INSERT INTO authors (name) VALUES (author)")
>
> but i don't see any code for creating the authors table
>
> also did you mean "name = author" instead of "name = :author"
>
> 3rd thing is that since you are using SQLALchemy, you should not be using
> pure SQL
>
> What i mean by models is something like this:
> https://github.com/Abdur-rahmaanJ/shopyo/blob/dev/shopyo/modules/product/models.py
>
> where you define code and your table is created the link uses
> flask-sqlalchemy but pure sqlalchemy should
> be somewhat similar
>
> 4th thing: when using
>
> for isbn, title, author, year in reader:
>
> without flask, do you actually see a value for author?
>
> Kind Regards,
>
> Abdur-Rahmaan Janhangeer
> compileralchemy  | blog
> 
> github 
> Mauritius
>
>
> On Thu, May 28, 2020 at 10:31 PM Buddy Peacock 
> wrote:
>
>> I'm not sure what you mean by models?  Am I missing something in my
>> environment perhaps?
>>
>> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
>> (920) 740-3411
>> linkedin.com/in/buddypeacock 
>>
>>
>>
>>
>> On Thu, May 28, 2020 at 2:26 PM Abdur-Rahmaan Janhangeer <
>> arj.pyt...@gmail.com> wrote:
>>
>>> Greetings,
>>>
>>>
>>> where did you define your models?
>>>
>>> Kind Regards,
>>>
>>>
>>> Abdur-Rahmaan Janhangeer
>>>
>>> https://www.github.com/Abdur-RahmaanJ
>>>
>>> Mauritius
>>>
>>> sent from gmail client on Android, that's why the signature is so ugly.
>>>
>>> On Thu, 28 May 2020, 22:22 Buddy Peacock, 
>>> wrote:
>>>
 Hello group,
 I have a pretty good background in MySQL & MSSQL as well as VB & Php,
 but I
 am new to Python and Postgresql.

 I am taking a class and working on a project to insert authors and books
 into a table.  My code for this is:
 ===
 import csv
 import os

 from flask import Flask
 from sqlalchemy import create_engine
 from sqlalchemy.orm import scoped_session, sessionmaker

 app = Flask(__name__)

 # Configure session to use filesystem
 app.config["SESSION_PERMANENT"] = False
 app.config["SESSION_TYPE"] = "filesystem"
 # Session(app)

 # Set up database
 engine = create_engine(os.getenv("DATABASE_URL"))
 db = scoped_session(sessionmaker(bind=engine))

 print (os.getenv("DATABASE_URL"))

 def main():
 f = open("books.csv")
 reader = csv.reader(f)
 for isbn, title, author, year in reader:
 if db.execute("SELECT * FROM authors WHERE name = :author",
 {"name": author}).rowcount == 0:
 db.execute("INSERT INTO authors (name) VALUES (author)")

 print(f" {author} was read.")
 db.commit()

 if __name__ == "__main__":
 main()
 
 If I comment out the "if" statement then the authors all print on
 screen.
 But when I run this with the if statement I am getting a message that
 says:
 ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
 value is required for bind parameter 'author'
 [SQL: SELECT * FROM authors WHERE name = %(author)s]
 [parameters: [{'name': 'author'}]]

 What am I missing?


 Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
 (920) 740-3411
 linkedin.com/in/buddypeacock 
 --
 https://mail.python.org/mailman/listinfo/python-list

>>>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Rich Shepard]

On Thu, 28 May 2020, Buddy Peacock wrote:


I'm not sure what you mean by models?  Am I missing something in my
environment perhaps?


The model (or models if you prefer) define the sqlalchemy classes that
reflect the structure and content of the postgrep table(s).

I put all classes (for all database tables) in a single module, model.py.

This assumes, of course, that you have defined the postgres database schema.

Regards,

Rich
--
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Abdur-Rahmaan Janhangeer]

You tried

for isbn, title, author, year in reader:

without flask, do you actually see a value for author?



Like using a normal print

Kind Regards,


Abdur-Rahmaan Janhangeer

https://www.github.com/Abdur-RahmaanJ

Mauritius

sent from gmail client on Android, that's why the signature is so ugly.

On Thu, 28 May 2020, 23:31 Buddy Peacock,  wrote:

> Thanks for your response Abdur-Rahmaan,
>
> The tables already exist and I have tried both  "name = author" and "name
> = :author"
>
> Regards,
>
> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
> (920) 740-3411
> linkedin.com/in/buddypeacock 
>
>
>
>
> On Thu, May 28, 2020 at 2:44 PM Abdur-Rahmaan Janhangeer <
> arj.pyt...@gmail.com> wrote:
>
>> Well,
>>
>>
>> See
>>
>> for isbn, title, author, year in reader:
>> if db.execute("SELECT * FROM authors WHERE name = :author",
>> {"name": author}).rowcount == 0:
>> db.execute("INSERT INTO authors (name) VALUES (author)")
>>
>> but i don't see any code for creating the authors table
>>
>> also did you mean "name = author" instead of "name = :author"
>>
>> 3rd thing is that since you are using SQLALchemy, you should not be using
>> pure SQL
>>
>> What i mean by models is something like this:
>> https://github.com/Abdur-rahmaanJ/shopyo/blob/dev/shopyo/modules/product/models.py
>>
>> where you define code and your table is created the link uses
>> flask-sqlalchemy but pure sqlalchemy should
>> be somewhat similar
>>
>> 4th thing: when using
>>
>> for isbn, title, author, year in reader:
>>
>> without flask, do you actually see a value for author?
>>
>> Kind Regards,
>>
>> Abdur-Rahmaan Janhangeer
>> compileralchemy  | blog
>> 
>> github 
>> Mauritius
>>
>>
>> On Thu, May 28, 2020 at 10:31 PM Buddy Peacock 
>> wrote:
>>
>>> I'm not sure what you mean by models?  Am I missing something in my
>>> environment perhaps?
>>>
>>> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
>>> (920) 740-3411
>>> linkedin.com/in/buddypeacock 
>>>
>>>
>>>
>>>
>>> On Thu, May 28, 2020 at 2:26 PM Abdur-Rahmaan Janhangeer <
>>> arj.pyt...@gmail.com> wrote:
>>>
 Greetings,


 where did you define your models?

 Kind Regards,


 Abdur-Rahmaan Janhangeer

 https://www.github.com/Abdur-RahmaanJ

 Mauritius

 sent from gmail client on Android, that's why the signature is so ugly.

 On Thu, 28 May 2020, 22:22 Buddy Peacock, 
 wrote:

> Hello group,
> I have a pretty good background in MySQL & MSSQL as well as VB & Php,
> but I
> am new to Python and Postgresql.
>
> I am taking a class and working on a project to insert authors and
> books
> into a table.  My code for this is:
> ===
> import csv
> import os
>
> from flask import Flask
> from sqlalchemy import create_engine
> from sqlalchemy.orm import scoped_session, sessionmaker
>
> app = Flask(__name__)
>
> # Configure session to use filesystem
> app.config["SESSION_PERMANENT"] = False
> app.config["SESSION_TYPE"] = "filesystem"
> # Session(app)
>
> # Set up database
> engine = create_engine(os.getenv("DATABASE_URL"))
> db = scoped_session(sessionmaker(bind=engine))
>
> print (os.getenv("DATABASE_URL"))
>
> def main():
> f = open("books.csv")
> reader = csv.reader(f)
> for isbn, title, author, year in reader:
> if db.execute("SELECT * FROM authors WHERE name = :author",
> {"name": author}).rowcount == 0:
> db.execute("INSERT INTO authors (name) VALUES (author)")
>
> print(f" {author} was read.")
> db.commit()
>
> if __name__ == "__main__":
> main()
> 
> If I comment out the "if" statement then the authors all print on
> screen.
> But when I run this with the if statement I am getting a message that
> says:
> ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
> value is required for bind parameter 'author'
> [SQL: SELECT * FROM authors WHERE name = %(author)s]
> [parameters: [{'name': 'author'}]]
>
> What am I missing?
>
>
> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
> (920) 740-3411
> linkedin.com/in/buddypeacock <
> https://www.linkedin.com/in/buddypeacock/>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why no camelCase in PEP 8?

[Peter J. Holzer]

On 2020-05-19 09:53:01 +0100, Rhodri James wrote:
> On 18/05/2020 22:07, Eli the Bearded wrote:
> > camelCase   ->   noCamelCase
> > snake_case  ->   no_snake_case
> > 
> > One of those is easier to "grep" for than the other.
> 
> Eh.  A changed case in the one, an extra character in the other; that's
> pretty much the same difficulty really.  I certainly don't find it "hard" to
> grep for _snake_case.

I think you misunderstood his argument. He is saying that by searching
for /snake_case/ you will find both "snake_case" and "no_snake_case".
But /camelCase/ won't match "noCamelCase" - you have to search for
/[cC]amelCase/ instead.

I'm not sure if I'm convinced by that. I prefer snake_case simply for
aesthetic reasons. My native language is German, so I should be used to
gratuitous capitalisation from early childhood - but I still find
camelCase ugly.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why no camelCase in PEP 8?

[Peter J. Holzer]

On 2020-05-19 05:59:30 +1000, Chris Angelico wrote:
> PEP 8 is a style guide for the Python standard library. It is the
> rules you must comply with if you are submitting a patch *to Python
> itself*. Nobody ever requires you to comply with it for any other
> code.

That's obviously not true: Many companies and projects have a coding
standard. Many of those coding standards will be based on or even
identical to PEP 8. And as an employee or contributor you may be
required to comply with it. Now you might argue that in this case you
aren't required to comply with PEP 8, but with the coding standard of
your company, but I would consider that excessive nitpickery.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy & Postgresql

[Buddy Peacock]

Yes, after commenting out all flask related lines and then running the
print, I do see all the authors.

Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
(920) 740-3411
linkedin.com/in/buddypeacock 




On Thu, May 28, 2020 at 3:48 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> You tried
>
> for isbn, title, author, year in reader:
>
> without flask, do you actually see a value for author?
>
>
>
> Like using a normal print
>
> Kind Regards,
>
>
> Abdur-Rahmaan Janhangeer
>
> https://www.github.com/Abdur-RahmaanJ
>
> Mauritius
>
> sent from gmail client on Android, that's why the signature is so ugly.
>
> On Thu, 28 May 2020, 23:31 Buddy Peacock,  wrote:
>
>> Thanks for your response Abdur-Rahmaan,
>>
>> The tables already exist and I have tried both  "name = author" and "name
>> = :author"
>>
>> Regards,
>>
>> Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
>> (920) 740-3411
>> linkedin.com/in/buddypeacock 
>>
>>
>>
>>
>> On Thu, May 28, 2020 at 2:44 PM Abdur-Rahmaan Janhangeer <
>> arj.pyt...@gmail.com> wrote:
>>
>>> Well,
>>>
>>>
>>> See
>>>
>>> for isbn, title, author, year in reader:
>>> if db.execute("SELECT * FROM authors WHERE name = :author",
>>> {"name": author}).rowcount == 0:
>>> db.execute("INSERT INTO authors (name) VALUES (author)")
>>>
>>> but i don't see any code for creating the authors table
>>>
>>> also did you mean "name = author" instead of "name = :author"
>>>
>>> 3rd thing is that since you are using SQLALchemy, you should not be
>>> using
>>> pure SQL
>>>
>>> What i mean by models is something like this:
>>> https://github.com/Abdur-rahmaanJ/shopyo/blob/dev/shopyo/modules/product/models.py
>>>
>>> where you define code and your table is created the link uses
>>> flask-sqlalchemy but pure sqlalchemy should
>>> be somewhat similar
>>>
>>> 4th thing: when using
>>>
>>> for isbn, title, author, year in reader:
>>>
>>> without flask, do you actually see a value for author?
>>>
>>> Kind Regards,
>>>
>>> Abdur-Rahmaan Janhangeer
>>> compileralchemy  | blog
>>> 
>>> github 
>>> Mauritius
>>>
>>>
>>> On Thu, May 28, 2020 at 10:31 PM Buddy Peacock 
>>> wrote:
>>>
 I'm not sure what you mean by models?  Am I missing something in my
 environment perhaps?

 Al (Buddy) Peacock, PMP, MCCT,  ITILv3, SMC, CSM, SPOC
 (920) 740-3411
 linkedin.com/in/buddypeacock
 




 On Thu, May 28, 2020 at 2:26 PM Abdur-Rahmaan Janhangeer <
 arj.pyt...@gmail.com> wrote:

> Greetings,
>
>
> where did you define your models?
>
> Kind Regards,
>
>
> Abdur-Rahmaan Janhangeer
>
> https://www.github.com/Abdur-RahmaanJ
>
> Mauritius
>
> sent from gmail client on Android, that's why the signature is so ugly.
>
> On Thu, 28 May 2020, 22:22 Buddy Peacock, 
> wrote:
>
>> Hello group,
>> I have a pretty good background in MySQL & MSSQL as well as VB & Php,
>> but I
>> am new to Python and Postgresql.
>>
>> I am taking a class and working on a project to insert authors and
>> books
>> into a table.  My code for this is:
>> ===
>> import csv
>> import os
>>
>> from flask import Flask
>> from sqlalchemy import create_engine
>> from sqlalchemy.orm import scoped_session, sessionmaker
>>
>> app = Flask(__name__)
>>
>> # Configure session to use filesystem
>> app.config["SESSION_PERMANENT"] = False
>> app.config["SESSION_TYPE"] = "filesystem"
>> # Session(app)
>>
>> # Set up database
>> engine = create_engine(os.getenv("DATABASE_URL"))
>> db = scoped_session(sessionmaker(bind=engine))
>>
>> print (os.getenv("DATABASE_URL"))
>>
>> def main():
>> f = open("books.csv")
>> reader = csv.reader(f)
>> for isbn, title, author, year in reader:
>> if db.execute("SELECT * FROM authors WHERE name = :author",
>> {"name": author}).rowcount == 0:
>> db.execute("INSERT INTO authors (name) VALUES (author)")
>>
>> print(f" {author} was read.")
>> db.commit()
>>
>> if __name__ == "__main__":
>> main()
>> 
>> If I comment out the "if" statement then the authors all print on
>> screen.
>> But when I run this with the if statement I am getting a message that
>> says:
>> ssqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A
>> value is required for bind parameter 'author'
>> [SQL: SELECT * FROM authors WHERE name = %(author)s]
>> [parameters: [{'name': 'author

Re: why no camelCase in PEP 8?

[Chris Angelico]

On Fri, May 29, 2020 at 6:20 AM Peter J. Holzer  wrote:
>
> On 2020-05-19 05:59:30 +1000, Chris Angelico wrote:
> > PEP 8 is a style guide for the Python standard library. It is the
> > rules you must comply with if you are submitting a patch *to Python
> > itself*. Nobody ever requires you to comply with it for any other
> > code.
>
> That's obviously not true: Many companies and projects have a coding
> standard. Many of those coding standards will be based on or even
> identical to PEP 8. And as an employee or contributor you may be
> required to comply with it. Now you might argue that in this case you
> aren't required to comply with PEP 8, but with the coding standard of
> your company, but I would consider that excessive nitpickery.
>

The OP said:
> My preference for using camelCase (in PEP 8, AKA mixedCase) is putting me at 
> odds with my colleagues, who point to PEP 8 as "the rules".
>

This smells like the incredibly strong misconception that PEP 8 needs
to govern every line of Python code ever written, or else it's "bad
code". This thread wouldn't have been started if it had been any other
style guide that the company had been chosen, because then it's
obvious that the choice is the company's. It's only when PEP 8 is
considered to be some sort of universal standard that we get this kind
of discussion.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system

[Sam]

On 5/28/20 12:49 AM, Rahul Gupta wrote:


I am having a Ubuntu system which has 125 Gb of RAM. I executed few python 
scripts on that system. Those scripts uses numpy arrays and pandas. Now 
execution was over but still 50 gb of RAM and 2 Gb cache and 8.4 Gb of swap is 
occupied. At this moment nothing is running on the system. I have googled it. 
Most of th result shows that python garbage collector is poor in performance. I 
want this memory to be cleaned and re claim. One of the easiest way is to 
restart the system but i dont want to restart i want a way to do this when the 
system is up and running. Kindly tell me how to do this. Thanks




Give us the output of:
cat /proc/meminfo
and
ps aux

Regards,
--
https://mail.python.org/mailman/listinfo/python-list

Re: creating a table within python code

[Peter J. Holzer]

On 2020-05-22 10:22:40 -0400, Buddy Peacock wrote:
> Thank you Souvik, but still having issues.  I have pasted the command line
> interaction this time.  My prior message is what appeared in the browser.
> 
> c:\Harvard\project1>SET FLASK_APP="create_db.py"
> 
> c:\Harvard\project1>set DATABASE_URL=postgres://
> guaqyugfujbudc:79ae65a6d8966991694906e4b96f20ebcfde5b80fb334e99d79d9300dd6ef...@ec2-34-200-72-77.compute-1.amazonaws.com:5432/dep18tfh5g2eme
> 
> c:\Harvard\project1>flask run
>  * Serving Flask app ""create_db.py"" (lazy loading)
   
[...]
> ModuleNotFoundError: No module named '"create_db'
   

I don't think those quotes were a good idea.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Peter J. Holzer]

On 2020-05-23 13:22:26 -0600, Mats Wichmann wrote:
> On 5/23/20 12:23 AM, Adam Preble wrote:
> > I wanted to update from 3.6.8 on Windows without necessarily moving
> > on to 3.7+ (yet), so I thought I'd try 3.6.9 or 3.6.10. 
> > 
> > All I see for both are source archives:
[...]
> 
> During the early part of a release cycle, installers are built.  Once
> the cycle moves into security fix-only mode, installers are not built.
> That's all you are seeing.

This seems a rather odd policy to me. Distributing a security fix in
source-only form will prevent many people from applying it (especially
on Windows).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Grant Edwards]

On 2020-05-27, Roel Schroeven  wrote:

> I find no hints of adding a backslash at the end to indicate directories.

I suspect that, like Unix, that's not an OS/filesystem thing but
merely a convention used by some user space applications to allow the
user to provide an additional hint as to his intention.

--
Grant







-- 
https://mail.python.org/mailman/listinfo/python-list

Re: why no camelCase in PEP 8?

[Roel Schroeven]

Peter J. Holzer schreef op 28/05/2020 om 22:09:

On 2020-05-19 09:53:01 +0100, Rhodri James wrote:

On 18/05/2020 22:07, Eli the Bearded wrote:

camelCase   ->noCamelCase
snake_case  ->no_snake_case

One of those is easier to "grep" for than the other.


Eh.  A changed case in the one, an extra character in the other; that's
pretty much the same difficulty really.  I certainly don't find it "hard" to
grep for _snake_case.


I think you misunderstood his argument. He is saying that by searching
for /snake_case/ you will find both "snake_case" and "no_snake_case".
But /camelCase/ won't match "noCamelCase" - you have to search for
/[cC]amelCase/ instead.

I'm not sure if I'm convinced by that. I prefer snake_case simply for
aesthetic reasons. My native language is German, so I should be used to
gratuitous capitalisation from early childhood - but I still find
camelCase ugly.


For me, the reason for disliking camelCase is consistency. I don't like 
how in camelCase words are written differently just because they appear 
at the front or not at the front. I don't like that the c of camelCase 
changes to the C in noCamelCase; not for the easy of searching (though 
that is an argument, just not the one that irks me most) but because 
it's not consistent. Things should change when their meaning changes, 
not when their place changes.


Both PascalCase and snake_case don't have that inconsistency, which is 
why I like them a lot better than camelCase.


There's another reason why I don't like camelCase: it smells like Java, 
and I don't like Java. That's an entirely irrational feeling though, of 
course.


--
"Honest criticism is hard to take, particularly from a relative, a
friend, an acquaintance, or a stranger."
-- Franklin P. Jones

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Terry Reedy]

On 5/28/2020 9:19 AM, Eryk Sun wrote:

On 5/26/20, BlindAnagram  wrote:


But if I try to make the directory myself (as I tried first):

   join(base, '..\\..\\', 'build', '\\')

I obtain:

'C:\\'

The documentation says that an absolute path in the parameter list for
join will discard all previous parameters but '\\' is not an absoute path!


First, to be clear, a rooted path in Windows is not absolute (*), but
that's not relevant here. What happens is that joining a path is like
issuing successive "cd" commands in the shell, and "cd \" takes you to
the root path of the current drive. The implementation of
ntpath.join() similarly tracks the current drive via
ntpath.splitdrive() while joining components.


Ah.  cd only changes the cwd within a device. Changing the active device 
is done otherwise.  In Command Prompt


C:\Users\Terry>f: # : required

F:\>c:

C:\Users\Terry>cd f:
F:\

C:\Users\Terry>cd f:/dev

C:\Users\Terry>f:

f:\dev>

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Roel Schroeven]

Eryk Sun schreef op 28/05/2020 om 15:51:

On 5/27/20, Chris Angelico  wrote:

On Thu, May 28, 2020 at 7:07 AM BlindAnagram 
wrote:

You can define a path however you want but it won't change the fact that
on Windows a path that ends in '\\' is inherently a path to a directory.


Citation needed.


See [MS-FSA] 2.1.5.1 Server Requests an Open of a File [1]. Here are
the relevant statements:

 The operation MUST be failed with STATUS_OBJECT_NAME_INVALID under
 any of the following conditions:
 
   * If PathName contains a trailing backslash and
 CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE.


i.e. trailing backslashes are not allowed for regular files


 
 If PathName contains a trailing backslash:
   * If StreamTypeToOpen is DataStream or
 CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE, the operation MUST
 be failed with STATUS_OBJECT_NAME_INVALID


i.e. again, trailing backslashes are not allowed for non-directory files.

There is no limitation listed that says something like "The operation 
must be failed if pathname doesn't contain a trailing backslash and 
CreateOptions.FILE_NON_DIRECTOR_FILE is FALSE". Path names referring to 
directories are allowed to have trailing backslashes, but no requirement 
to do so. Directory PathNames with and without trailing backslashes are 
handled exactly the same.



Internally, WinAPI CreateFileW calls NTAPI NtCreateFile with the
create option FILE_NON_DIRECTORY_FILE (i.e. only open or create a data
file stream), unless backup semantics are requested in order to be
able to open a directory (i.e. an index stream), in which case the
call uses neither FILE_NON_DIRECTORY_FILE nor FILE_DIRECTORY_FILE and
leaves it up to the path name.


No, that is not my understanding. It is up to the actual type of file 
specified by the path. CreateFileW using FILE_FLAG_BACKUP_SEMANTICS  can 
only open existing directories, so there is no need to look at the last 
character of the path.



In other words, with backup semantics, if you need to ensure that
only a directory is opened, add a trailing slash.
It's not like you have some random path, you think "hey, I only want to 
open this if it is a directory", and then do


CreateFileW(
  L"...\\...\\",
  ..,
  ...,
  ...,
  ...,
  FILE_FLAG_BACKUP_SEMANTICS,
  NULL);

What happens instead is that you have a path to a real existing object 
(CreateFileW on directories works only with both OPEN_EXISTING and 
FILE_FLAG_BACKUP_SEMANTICS specified: you can not create a directory 
using CreateFileW), and CreateFileW (via NtCreateFile) will open it as a 
directory if the object is a directory, and as a file it is a file. 
Exception is that it will refuse to open it as a file if there's a 
backslash in the end, while for directories backslashes are optional. 
But if you never add a backslash in the end, everything will work just 
fine for both files and directories.


--
"Honest criticism is hard to take, particularly from a relative, a
friend, an acquaintance, or a stranger."
-- Franklin P. Jones

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Roel Schroeven]

BlindAnagram schreef op 28/05/2020 om 11:00:
On 27/05/2020 23:39, Roel Schroeven wrote: 

I find no hints of adding a backslash at the end to indicate directories.
If you can point me to convincing evidence in the documentation I'll
change my mind.


And if you find a counterexample, I will change mine.


It's hard to prove a negative.

The only thing I can find is that there are cases where doing operations 
on files fail if there's a backslash at the end, whereas operations on 
directories do work with a backslash at the end; but those also work 
without the backslash. Even then, the distinction between what operation 
is done is never decided by the backslash; only if it is decided that 
you're doing something file-related and there is a backslash, then it 
fails. It's not like it turns into a directory-related operation. Ergo, 
adding backslashes practically accomplishes nothing.


What really defines what kind of operation will be done, is
- which function you call
- or the flags you pass to the function
- or for some operation when working with existing objects, the type of 
those existing objects


Also, if we're going to look in the other direction: we can ask Windows 
for information about objects on disk, and it will tell use the type 
using flags, not by adding backslashes. We can use e.g. FindFirstFile / 
FindNextFile / FindClose. dwFileAttributes in the WIN32_FIND_DATA 
structure will have FILE_ATTRIBUTE_DIRECTORY set for directories and not 
for normal files. cFileName won't have a trailing backslash. If the 
semantics were as you say, I'd expect Windows to at least using them itself.


In summary, IMO:

- Windows allows trailing backslashes for directories.
- Windows doesn't allow trailing backslashes for files.
- Windows doesn't use trailing backslashes for distinction between files 
and directories.

- And so I don't see how/why adding a backslash makes any real difference.

--
"Honest criticism is hard to take, particularly from a relative, a
friend, an acquaintance, or a stranger."
-- Franklin P. Jones

Roel Schroeven

--
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system

[Peter J. Holzer]

On 2020-05-27 22:49:53 -0700, Rahul Gupta wrote:
> I am having a Ubuntu system which has 125 Gb of RAM. I executed few
> python scripts on that system. Those scripts uses numpy arrays and
> pandas. Now execution was over but still 50 gb of RAM and 2 Gb cache
> and 8.4 Gb of swap is occupied. At this moment nothing is running on
> the system. I have googled it. Most of th result shows that python
> garbage collector is poor in performance. I want this memory to be
> cleaned and re claim.

As others have already pointed out, a process which doesn't exist can't
use memory (on Linux or any other sane OS). So either python is still
running or something else is using the memory.

To find out what is using the memory, run the command
top
in a terminal. Then press Shift-M to sort processes by memory
consumption. The offenders should now be at the top.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Constructing mime image attachment

[DL Neil via Python-list]

On 28/05/20 11:48 PM, Joseph L. Casale wrote:

I have some json encoded input for nodemailer 
(https://nodemailer.com/message/embedded-images)
where the path key is a string value which contains the base64 encoded data 
such as:

{
 html: 'Embedded image: ',
 attachments: [{
 filename: 'image.png',
 path: 'data:image/png;base64,iVBORw',
 cid: 'uni...@nodemailer.com'
 }]
}

Does an approach exist to construct a MIMEImage object with the existing
base64 string without parsing it and decoding it?



Might the Python Standard Library modules for Internet Handling offer 
suitable facilities, eg base64 or mimetypes?

https://docs.python.org/3/library/netdata.html
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: why no camelCase in PEP 8?

[Peter J. Holzer]

On 2020-05-29 06:27:31 +1000, Chris Angelico wrote:
> On Fri, May 29, 2020 at 6:20 AM Peter J. Holzer  wrote:
> > On 2020-05-19 05:59:30 +1000, Chris Angelico wrote:
> > > PEP 8 is a style guide for the Python standard library. It is the
> > > rules you must comply with if you are submitting a patch *to Python
> > > itself*. Nobody ever requires you to comply with it for any other
> > > code.
> >
> > That's obviously not true: Many companies and projects have a coding
> > standard. Many of those coding standards will be based on or even
> > identical to PEP 8. And as an employee or contributor you may be
> > required to comply with it. Now you might argue that in this case you
> > aren't required to comply with PEP 8, but with the coding standard of
> > your company, but I would consider that excessive nitpickery.
> >
> 
> The OP said:
> > My preference for using camelCase (in PEP 8, AKA mixedCase) is
> > putting me at odds with my colleagues, who point to PEP 8 as "the
> > rules".
> >
> 
> This smells like the incredibly strong misconception that PEP 8 needs
> to govern every line of Python code ever written, or else it's "bad
> code". This thread wouldn't have been started if it had been any other
> style guide that the company had been chosen, because then it's
> obvious that the choice is the company's. It's only when PEP 8 is
> considered to be some sort of universal standard that we get this kind
> of discussion.

It may be the case in this specific instance (although I don't know, as
I don't work for the OP's company and don't know his colleagues). I
don't think it is generally the case.

When a company introduces a coding standard, I think "just use PEP 8" is
a very reasonable choice (it's not inherently better or worse than any
other style, but just by being familiar to a lot of Python programmers
it is likely to elicit less resistance and bike-shedding).

So this might lead to the following exchange during a code review:

A: Please don't use camelCase for variables. It's against our coding
style.

B: Why don't we allow camelCase for variables?

A: Because PEP 8 doesn't allow it.

B: Why doesn't PEP 8 allow it?

A: Uh, ask the PEP 8 authors.

At which point B becomes the OP of this thread.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[MRAB]

On 2020-05-28 22:34, Terry Reedy wrote:

On 5/28/2020 9:19 AM, Eryk Sun wrote:

On 5/26/20, BlindAnagram  wrote:


But if I try to make the directory myself (as I tried first):

   join(base, '..\\..\\', 'build', '\\')

I obtain:

'C:\\'

The documentation says that an absolute path in the parameter list for
join will discard all previous parameters but '\\' is not an absoute path!


First, to be clear, a rooted path in Windows is not absolute (*), but
that's not relevant here. What happens is that joining a path is like
issuing successive "cd" commands in the shell, and "cd \" takes you to
the root path of the current drive. The implementation of
ntpath.join() similarly tracks the current drive via
ntpath.splitdrive() while joining components.


Ah.  cd only changes the cwd within a device. Changing the active device
is done otherwise.  In Command Prompt

C:\Users\Terry>f: # : required

F:\>c:

C:\Users\Terry>cd f:
F:\

C:\Users\Terry>cd f:/dev

C:\Users\Terry>f:

f:\dev>


cd has a switch /d that will also change the drive:

C:\Users\Terry>cd /d f:/dev

f:\dev>
--
https://mail.python.org/mailman/listinfo/python-list

Re: why no camelCase in PEP 8?

[Terry Reedy]

On 5/28/2020 4:18 PM, Peter J. Holzer wrote:

On 2020-05-19 05:59:30 +1000, Chris Angelico wrote:

PEP 8 is a style guide for the Python standard library. It is the
rules you must comply with if you are submitting a patch *to Python
itself*. Nobody ever requires you to comply with it for any other
code.


That's obviously not true: Many companies and projects have a coding
standard. Many of those coding standards will be based on or even
identical to PEP 8. And as an employee or contributor you may be
required to comply with it.


Revise Chris' claim to "Neither the PSF nor the Python core developers 
require* that owners of non-stdlib code comply with PEP 8" and it would 
be true.


* We could have, by baking it into the language, by making uppercase 
following lowercase in identifiers illegal.  But that would disabled 
part of the stdlib.



Now you might argue that in this case you
aren't required to comply with PEP 8, but with the coding standard of
your company, but I would consider that excessive nitpickery.


I don't.  If an entity with a police force adopts part of the Model Fire 
Code written by an International Association of Fire Marshals (or 
whatever), it matters that any enforcement is done by that state's 
officials rather toothless disapproval of the association.


In any case, Guido as BDFL did not enforce the function-name rule, as 
illustrated by unittest (I would have prefered otherwise).  Moveover, he 
made it clear in PEP 8 that strict enforcement of its rules was not one 
of its rules.  Many of the rules explicitly allow for exceptions in 
exceptional circumstances.  So required no-exception compliance is an 
add-on by other entities.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Behaviour of os.path.join

[Eryk Sun]

On 5/28/20, Roel Schroeven  wrote:
> Eryk Sun schreef op 28/05/2020 om 15:51:
>> On 5/27/20, Chris Angelico  wrote:
>>> On Thu, May 28, 2020 at 7:07 AM BlindAnagram 
>>> wrote:
 You can define a path however you want but it won't change the fact
 that on Windows a path that ends in '\\' is inherently a path to a
 directory.

[snip]

> i.e. again, trailing backslashes are not allowed for non-directory files.
>
> There is no limitation listed that says something like "The operation
> must be failed if pathname doesn't contain a trailing backslash and
> CreateOptions.FILE_NON_DIRECTOR_FILE is FALSE". Path names referring to
> directories are allowed to have trailing backslashes, but no requirement
> to do so. Directory PathNames with and without trailing backslashes are
> handled exactly the same.

The statement I was documenting (quoted above) is that a path that
ends in a slash has to be a directory, not that a directory cannot be
accessed without a trailing slash.

The create option for NtCreateFile can be FILE_NON_DIRECTORY_FILE,
FILE_DIRECTORY_FILE, or left undefined. If it's not defined, and the
opened path has a trailing slash, then for an open-existing
disposition, the existing stream must be a directory.  It's an
invalid-name error (STATUS_OBJECT_NAME_INVALID) if the existing stream
is a regular file. If FILE_NON_DIRECTORY_FILE is specified (the
default for WinAPI CreateFileW), the path cannot have a trailing slash
because that designates a directory, which is inconsistent with the
create option. Again, this is an invalid-name error, which is distinct
from errors where the create option disagrees with the stream type
(i.e. STATUS_FILE_IS_A_DIRECTORY and STATUS_NOT_A_DIRECTORY).

>> Internally, WinAPI CreateFileW calls NTAPI NtCreateFile with the
>> create option FILE_NON_DIRECTORY_FILE (i.e. only open or create a data
>> file stream), unless backup semantics are requested in order to be
>> able to open a directory (i.e. an index stream), in which case the
>> call uses neither FILE_NON_DIRECTORY_FILE nor FILE_DIRECTORY_FILE and
>> leaves it up to the path name.
>
> No, that is not my understanding. It is up to the actual type of file
> specified by the path. CreateFileW using FILE_FLAG_BACKUP_SEMANTICS  can
> only open existing directories, so there is no need to look at the last
> character of the path.

It is true that just including a trailing slash does not set the
StreamTypeToOpen to a directory stream. That's why a mismatch leads to
an invalid-name error instead of a directory error (i.e.
STATUS_NOT_A_DIRECTORY), because it's only the name in the opened path
that's inconsistent with the existing stream type.

In order to create a directory, the StreamTypeToOpen must be a
directory. One way to set that is to explicitly use the
FILE_DIRECTORY_FILE create option. This is possible by using a
CREATE_NEW disposition with the flags and attributes
FILE_ATTRIBUTE_DIRECTORY | FILE_FLAG_BACKUP_SEMANTICS |
FILE_FLAG_POSIX_SEMANTICS. For example:

>>> flags = FILE_ATTRIBUTE_DIRECTORY
>>> flags |= FILE_FLAG_BACKUP_SEMANTICS
>>> flags |= FILE_FLAG_POSIX_SEMANTICS
>>> disposition = CREATE_NEW
>>> h = CreateFile('spam', 0, 0, None, disposition, flags, None)
>>> os.path.isdir('spam')
True

AFAIK, Microsoft never documented the above capability, so, even
though NT's implementation of the Windows API has included it for
almost 30 years, it's still reasonable to ignore it.

As documented in [MS-FSA], the other way to set StreamTypeToOpen to a
directory is to explicitly open an $INDEX_ALLOCATION stream (named
"$I30", but the name can be omitted). This requires the filesystem to
support file streams (e.g. NTFS, ReFS).  For example:

>>> flags = 0
>>> disposition = CREATE_NEW
>>> h = CreateFile('eggs::$INDEX_ALLOCATION', 0, 0, None,
disposition, flags, None)
>>> os.path.isdir('eggs')
True

Notice that I didn't have to request backup semantics. Thus it created
a directory even though the default FILE_NON_DIRECTORY_FILE create
option was used. That's peculiar, but it's actually documented by
[MS-FSA]:

* If CreateOptions.FILE_DIRECTORY_FILE is TRUE then StreamTypeToOpen =
  DirectoryStream.

* Else if StreamTypeNameToOpen is "$INDEX_ALLOCATION" then
  StreamTypeToOpen = DirectoryStream.

* Else if CreateOptions.FILE_NON_DIRECTORY_FILE is FALSE,
  StreamNameToOpen is empty, StreamTypeNameToOpen is empty, Open.File
  is not NULL, and Open.File.FileType is DirectoryFile then
  StreamTypeToOpen = DirectoryStream.

* Else StreamTypeToOpen = DataStream.

> But if you never add a backslash in the end, everything will work just
> fine for both files and directories.

If you expect to open a directory, then appending a trailing slash and
relying on the the invalid-name error is a one way to ensure that,
though the error isn't really specific enough for my liking. To be
more explicit, if programming at a lower level in C/C++, query
GetF

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Terry Reedy]

On 5/28/2020 5:20 PM, Peter J. Holzer wrote:

On 2020-05-23 13:22:26 -0600, Mats Wichmann wrote:

On 5/23/20 12:23 AM, Adam Preble wrote:

I wanted to update from 3.6.8 on Windows without necessarily moving
on to 3.7+ (yet), so I thought I'd try 3.6.9 or 3.6.10.

All I see for both are source archives:



During the early part of a release cycle, installers are built.


Only for Windows and now for macOS.  Python.org only ever distributes 
source archives for *nix.  Distributors can add binaries to their 
package system.



 Once
the cycle moves into security fix-only mode, installers are not built.


We continue to apply security fixes for the benefit of server operators 
who are slow to upgrade and who want minimal change -- only those that 
they really need.  We make security-fix releases primarily for the 
benefit of *nix distributors who want to update their x.y package, but 
not for every x.y commit.  It also give a periodic new name for Python 
x.y with a new batch of fixes.



This seems a rather odd policy to me.


Not if one considers the intended users.
Do you prefer we not make these releases?

Anyone running servers on Windows should have Visual Studio and git 
installed as they should be able to compile their own binaries.  Anyone 
with control of their machine (so that they can download and install 
things) can install VS and git with the instructions in 
devguide.python.org. At that point, clone python/cpython and run 
PCbuild\build.bat -e (to build external dependencies) and maybe add 
other options, and python(_d).exe will appear in PCbuild\win32.



Distributing a security fix in
source-only form will prevent many people from applying it (especially
on Windows).


Nearly all bug fixes considered to be security risk fixes are first 
applied to master (the 'next' version), then maintenance versions, which 
do get installers, and only then to old security-fix versions.  The 
latter take extra effort as they are less likely to automatically 
backport, and on Windows, older versions run on more Windows versions.


The OP is so far choosing to not use an installer with those fixes.  By 
not doing so, he is missing out on the maybe 2000 non-security fixes and 
some enhancements that likely would benefit him more than maybe 50 
mostly obscure fixes added between 3.6.8 and 3.6.10*.  If a rare user 
such as Adam also chooses to not compile the latter, that is his choice.


*In the last 12 months, the ratio of fixed security issues to all fixed 
issues is 51/2087 = 2.4%, and for 5 years, 112/7825 = 1.4%.  There are 
68 open security issues, some of which will be closed other than as 'fixed'.


Source only releases only block Windows/Mac users who choose not to 
upgrade to a released installer and who cannot or choose not to compile.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Mike Dewhirst]

On 29/05/2020 10:51 am, Terry Reedy wrote:

On 5/28/2020 5:20 PM, Peter J. Holzer wrote:

On 2020-05-23 13:22:26 -0600, Mats Wichmann wrote:

On 5/23/20 12:23 AM, Adam Preble wrote:

I wanted to update from 3.6.8 on Windows without necessarily moving
on to 3.7+ (yet), so I thought I'd try 3.6.9 or 3.6.10.

All I see for both are source archives:



During the early part of a release cycle, installers are built.


Only for Windows and now for macOS.  Python.org only ever distributes 
source archives for *nix.  Distributors can add binaries to their 
package system.



 Once
the cycle moves into security fix-only mode, installers are not built.


We continue to apply security fixes for the benefit of server 
operators who are slow to upgrade and who want minimal change -- only 
those that they really need.  We make security-fix releases primarily 
for the benefit of *nix distributors who want to update their x.y 
package, but not for every x.y commit.  It also give a periodic new 
name for Python x.y with a new batch of fixes.



This seems a rather odd policy to me.


Not if one considers the intended users.
Do you prefer we not make these releases?

Anyone running servers on Windows should have Visual Studio and git 
installed as they should be able to compile their own binaries.  
Anyone with control of their machine (so that they can download and 
install things) can install VS and git with the instructions in 
devguide.python.org. At that point, clone python/cpython and run 
PCbuild\build.bat -e (to build external dependencies) and maybe add 
other options, and python(_d).exe will appear in PCbuild\win32.



Distributing a security fix in
source-only form will prevent many people from applying it (especially
on Windows).


Nearly all bug fixes considered to be security risk fixes are first 
applied to master (the 'next' version), then maintenance versions, 
which do get installers, and only then to old security-fix versions.  
The latter take extra effort as they are less likely to automatically 
backport, and on Windows, older versions run on more Windows versions.


The OP is so far choosing to not use an installer with those fixes.  
By not doing so, he is missing out on the maybe 2000 non-security 
fixes and some enhancements that likely would benefit him more than 
maybe 50 mostly obscure fixes added between 3.6.8 and 3.6.10*.  If a 
rare user such as Adam also chooses to not compile the latter, that is 
his choice.


*In the last 12 months, the ratio of fixed security issues to all 
fixed issues is 51/2087 = 2.4%, and for 5 years, 112/7825 = 1.4%. 
There are 68 open security issues, some of which will be closed other 
than as 'fixed'.


Source only releases only block Windows/Mac users who choose not to 
upgrade to a released installer and who cannot or choose not to compile.


I am an example

I installed all the Pythons on my Windows 10 dev machine (locked into 
Windows by having clients) but I'm also locked into Python 3.6.9 on my 
Ubuntu 18.04 production machines.


After chasing down an obscure problem I decided to go back to Py36 on 
Windows to be using the same versions in dev as in prd. I couldn't find 
an installer on python.org so I retrieved one (3.6.5) from my archives.


I choose to avoid Visual Studio and I won't bother with cygwin any more 
after some pain a decade or so ago. Therefore I choose not to compile.


For me it won't be long before I can upgrade my production machines to 
20.04 and whatever Python3 comes with that and all will be well.


If I was asked to suggest a guide for which versions ought to get a 
Windows binary I would look at the most popular LTS *nix distros and 
keep Windows binaries in step just to support people like me who cannot 
live with too much Windows clutter. Think of it as deeply humanitarian 
generosity.


Honestly, if you let it, Windows just absolutely knows what you really 
meant despite what you tell it. It is a necessary evil when your clients 
use it.


Cheers

Mike

--
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system (rmlibre)

[rmlibre]

We just ran into this problem when running our aiootp package's memory
hard password hashing function (https://github.com/rmlibre/aiootp/). The
memory was not being cleared after the function finished running but the
script was still live. We tried making sure everything went out of scope
and deleting things explicitly inside the function, but that didn't
help. We even tried forcing the garbage collector to free up
unreferenced memory with import gc; gc.collect(). But that only had a
small dent in the memory that was being built up.

The most useful answer online had to do with Python's free lists being
created automatically when very large datasets were being processed
(https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616).


After putting the memory intensive work into a separate process, as the
answer suggested, the memory was finally freed after execution.

In our case, we wound up passing the result back in a
``multiprocessing.Manager().List()``. Though, passing your whole numpy
dataset back doesn't seem feasible. I'd recommend doing the necessary
memory intensive work in a separate process and passing only the
necessary conclusions back to the main process. 

Or you could set up a multiprocessing queue to pass control messages to
the spawned (daemon) process so it returns desired results on demand. 


Here's an excerpt from our code after the fix in case it's helpful.

>@classmethod
>def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256):
>"""
>The ``passcrypt`` function can be highly memory intensive.
>These resources may not be freed up, & often are not, because of
>python quirks around memory management. This is a huge problem.
>So to force the release of those resources, we run the function
>in another process which is guaranteed to release them.
>"""
>cls._validate_passcrypt_args(kb, cpu, hardness)
>state = Manager().list()
>process = Process(
>target=cls._passcrypt,
>args=(password, salt),
>kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state),
>)
>process.start()
>process.join()
>return state.pop()




On 2020-05-28 16:00, python-list-requ...@python.org wrote:
> Send Python-list mailing list submissions to
>   python-list@python.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>   https://mail.python.org/mailman/listinfo/python-list
> or, via email, send a message with subject or body 'help' to
>   python-list-requ...@python.org
> 
> You can reach the person managing the list at
>   python-list-ow...@python.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-list digest..."
> 
> Today's Topics:
> 
>1. Re: Custom logging function (zljubi...@gmail.com)
>2. Ram memory not freed after executing python script on ubuntu
>   system (Rahul Gupta)
>3. Re: Ram memory not freed after executing python script on
>   ubuntu system (Chris Angelico)
>4. Re: Custom logging function (Peter Otten)
>5. Re: Elegant hack or gross hack? TextWrapper and escape codes
>   (Peter Otten)
>6. Re: Elegant hack or gross hack? TextWrapper and escape codes
>   (Chris Angelico)
>7. Re: Elegant hack or gross hack? TextWrapper and escape codes
>   (Peter Otten)
>8. Re: Ram memory not freed after executing python script on
>   ubuntu system (Rahul Gupta)
>9. Re: Ram memory not freed after executing python script on
>   ubuntu system (Chris Angelico)
>   10. Re: Behaviour of os.path.join (BlindAnagram)
>   11. Constructing mime image attachment (Joseph L. Casale)
>   12. Re: Behaviour of os.path.join (Eryk Sun)
>   13. Re: Behaviour of os.path.join (Eryk Sun)
>   14. Re: Behaviour of os.path.join (BlindAnagram)
>   15. Re: Behaviour of os.path.join (Eryk Sun)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Chris Angelico]

On Fri, May 29, 2020 at 11:57 AM Mike Dewhirst  wrote:
>
> I am an example
>
> I installed all the Pythons on my Windows 10 dev machine (locked into
> Windows by having clients) but I'm also locked into Python 3.6.9 on my
> Ubuntu 18.04 production machines.

Be careful of assuming too much here. The general policy with most
Linux distributions is to number the version according to the oldest
component in it, but they are free to backport whatever changes they
choose. I have Python 3.4.4 and 3.5.3 on this system (Debian Stretch),
but the 3.5 is numbered "3.5.3-1+deb9u1" which implies some collection
of additional patches. (I'd have to dig deep in the changelogs if I
cared exactly *which* patches.)

> After chasing down an obscure problem I decided to go back to Py36 on
> Windows to be using the same versions in dev as in prd. I couldn't find
> an installer on python.org so I retrieved one (3.6.5) from my archives.

That'd be the same feature version, but depending exactly what the
problem is, there might not be *any* Windows build that exactly
corresponds.

> If I was asked to suggest a guide for which versions ought to get a
> Windows binary I would look at the most popular LTS *nix distros and
> keep Windows binaries in step just to support people like me who cannot
> live with too much Windows clutter. Think of it as deeply humanitarian
> generosity.

Even if that were possible, who are you asking to do this? Whose time
is going to be put into finessing every point release to make sure
it's still buildable on Windows? "LTS" doesn't mean anything since
basically EVERY version of Python is in an LTS of Red Hat (they're
probably still shipping Python 2.3 somewhere).

> Honestly, if you let it, Windows just absolutely knows what you really
> meant despite what you tell it. It is a necessary evil when your clients
> use it.

If your clients use it, are you going to pay someone to build your
installers? And if you aren't going to pay, who is?

Find someone who'd be willing to maintain Windows binaries for you,
and see what they'd charge you for that. Then judge that against the
cost of dropping support for Python 3.6 and requiring your users to
upgrade to a fully-supported version.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system (rmlibre)

[Chris Angelico]

On Fri, May 29, 2020 at 12:08 PM  wrote:
>
>
> We just ran into this problem when running our aiootp package's memory
> hard password hashing function (https://github.com/rmlibre/aiootp/).

Have you considered implementing that module in something else? Try
Cythonizing it and see if suddenly your memory usage drops - not
because of garbage collection, but because you're no longer using
integer objects at all.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Adam Preble]

On Thursday, May 28, 2020 at 7:57:04 PM UTC-5, Terry Reedy wrote:
> The OP is so far choosing to not use an installer with those fixes.  By 
> not doing so, he is missing out on the maybe 2000 non-security fixes and 
> some enhancements that likely would benefit him more than maybe 50 
> mostly obscure fixes added between 3.6.8 and 3.6.10*.  If a rare user 
> such as Adam also chooses to not compile the latter, that is his choice.

I was going to just stay mute about why I was even looking at 3.6.10, but I
felt I should weigh in after some of the other responses. I think somebody
would find the issues interesting.

We had found what looked like a bug in the Python Launcher where it would
eat command line arguments meant for the script. I would find some stuff 
missing from sys.argv in a script that just imports sys and prints out sys.argv 
if I ran it directly in cmd.exe as "script.py." If I ran it as "python 
script.py" then everything was good as usual. So I figured while sorting out 
what was wrong that I should try the latest 3.6 interpreter since it would be a 
safe bet.

Our organization finally lifted Sisyphus' rock over the 2.7 hump earlier in the 
year by moving to 3.6. So imagine my surprise when I found the latest 3.6 
releases were just source tarballs. This left me with a dilemma and I'm still 
working through it. I haven't filed an issue about this because I haven't 
completed my own due diligence on the problem by trying it on a "latest."

For the sake of this particular problem, I think I can just use 3.8.3 for
exploration, but I'm worrying about my wider organization. I can't count on 3.8 
because of some module dependencies our organization's software. 3.7 has a 
similar issue. So I figured I'd actually just build the thing and see what I 
can do.

I did manage to build it, but there was surprisingly a few quirks. I caused
some of it. For example, I didn't care about most of the externals before,
but I made sure to include them if I was create a release for others. A few
thousand people would be using this and I'm the one that would be
accountable if it went bust. So I made sure all the major externals were
incorporated, and a lot of those were messing up. Generally, the externals
would download, but some would not get moved/renamed to their final name,
and then the build would fail when trying to find them. So I wound up with
an installation that seemed to run my own code just fine in trials, but I
would be terrified to post into it our organization's software stack.

I'm now concerned about how long we have with 3.6 because people clearly
want us to move on even beyond that. I look online and the official support
window for it ends at the end of next year, but it looks like the real
support window for that on Windows has already ended. So our organization
may have miscalculated this. What does that mean if we managed to make it
to 3.8 in a few months? We can't do it right now due to a few missing
modules, but now we have to question if we'll only get a year out of 3.8
before we're doing this all over again.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Chris Angelico]

On Fri, May 29, 2020 at 1:46 PM Adam Preble  wrote:
> I'm now concerned about how long we have with 3.6 because people clearly
> want us to move on even beyond that. I look online and the official support
> window for it ends at the end of next year, but it looks like the real
> support window for that on Windows has already ended. So our organization
> may have miscalculated this. What does that mean if we managed to make it
> to 3.8 in a few months? We can't do it right now due to a few missing
> modules, but now we have to question if we'll only get a year out of 3.8
> before we're doing this all over again.

Upgrading from 3.8 to 3.9 won't be hard. Check the release notes:

https://docs.python.org/3.9/whatsnew/3.9.html

In fact, even upgrading from 3.6 to 3.8 is unlikely to break anything.
Or go direct to 3.9, depending on how soon you're likely to complete
that transition (expected release date is October). Don't be afraid of
the minor upgrades.

I'm currently running a mix of 3.9 and 3.10, since there are a couple
of things that broke in 3.10. But that's not even into its first alpha
yet. By the time they get to release, generally most stuff won't break
:)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Dieter Maurer]

Mike Dewhirst wrote at 2020-5-29 11:55 +1000:
>On 29/05/2020 10:51 am, Terry Reedy wrote:
> ...
>> Source only releases only block Windows/Mac users who choose not to
>> upgrade to a released installer and who cannot or choose not to compile.
>
>I am an example
>
>I installed all the Pythons on my Windows 10 dev machine (locked into
>Windows by having clients) but I'm also locked into Python 3.6.9 on my
>Ubuntu 18.04 production machines.

You are not locked into Python 3.6 on Ubuntu: it is quite easy
to compile Python on *nix (this includes Ubuntu 18.04) yourself
(I did so recently for Python 3.9[a5]).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Mike Dewhirst]

On 29/05/2020 12:26 pm, Chris Angelico wrote:

On Fri, May 29, 2020 at 11:57 AM Mike Dewhirst  wrote:

I am an example

I installed all the Pythons on my Windows 10 dev machine (locked into
Windows by having clients) but I'm also locked into Python 3.6.9 on my
Ubuntu 18.04 production machines.

Be careful of assuming too much here. The general policy with most
Linux distributions is to number the version according to the oldest
component in it, but they are free to backport whatever changes they
choose. I have Python 3.4.4 and 3.5.3 on this system (Debian Stretch),
but the 3.5 is numbered "3.5.3-1+deb9u1" which implies some collection
of additional patches. (I'd have to dig deep in the changelogs if I
cared exactly *which* patches.)


After chasing down an obscure problem I decided to go back to Py36 on
Windows to be using the same versions in dev as in prd. I couldn't find
an installer on python.org so I retrieved one (3.6.5) from my archives.

That'd be the same feature version, but depending exactly what the
problem is, there might not be *any* Windows build that exactly
corresponds.


If I was asked to suggest a guide for which versions ought to get a
Windows binary I would look at the most popular LTS *nix distros and
keep Windows binaries in step just to support people like me who cannot
live with too much Windows clutter. Think of it as deeply humanitarian
generosity.

Even if that were possible, who are you asking to do this? Whose time
is going to be put into finessing every point release to make sure
it's still buildable on Windows?


I was careful to not ask anyone to do anything. "If I was asked ..."

Making sure something is buildable when it involves compiling on Windows 
is well beyond my pay-grade nowadays. I know just how hard it is. 
However, I would say (and I've said it before) Christoph Gohlke deserves 
a Nobel prize for his abovementioned deeply humanitarian generosity. 
Without him (and Mark Hammond) there would be very little Python dev 
done on Windows - if any!


The only reason I took up Python (after being burned by proprietary tool 
makers) was because it was promised to run on pretty much everything. 
Especially Apache platforms - which also run on Windows.



"LTS" doesn't mean anything since
basically EVERY version of Python is in an LTS of Red Hat (they're
probably still shipping Python 2.3 somewhere).


Honestly, if you let it, Windows just absolutely knows what you really
meant despite what you tell it. It is a necessary evil when your clients
use it.

If your clients use it,


They don't use Python. They use Windows and I need to use Windows to be 
able to support them in what they do. I install Apache on Windows if I 
need to.


It would be handy if Microsoft delivered Windows with Python installed 
but what can you expect from them if they can't see demand for it.




  are you going to pay someone to build your
installers? And if you aren't going to pay, who is?


I agree with your sentiment. But without specific funding I can't afford 
to pay anyone. I'll choose different solutions. But if I needed it badly 
enough I'd find the brainspace and do it myself.



Find someone who'd be willing to maintain Windows binaries for you,
and see what they'd charge you for that. Then judge that against the
cost of dropping support for Python 3.6 and requiring your users to
upgrade to a fully-supported version.


Chris, you are preaching to the converted. You won't get a 
counter-argument from me. I piped up in the first place because I saw 
the OP in trouble and felt it was necessary to say something. I thought 
my scenario might be interesting. As I said earlier it isn't critical 
for me.


There are lot of Python people working on Windows and they do so because 
they have no choice. Windows is simply the dominant platform. That's 
where most clients live.


All I'm saying (now) is I really hope PSF is supporting Christoph and 
Mark because they make it possible to use Python on Windows.


Cheers

Mike



ChrisA


--
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system (rmlibre)

[Dieter Maurer]

rmli...@riseup.net wrote at 2020-5-28 18:56 -0700:
>We just ran into this problem when running our aiootp package's memory
>hard password hashing function (https://github.com/rmlibre/aiootp/). The
>memory was not being cleared after the function finished running but the
>script was still live.

I hope you are aware that Python can do nothing to change this:
the elementary memory management feature provided by operating systems
is a way to increase or decrease the "heap" (there is a different
possibility (memory mapping), but it is efficient only for large chunks
of memory). Python cannot truncate the heap below the last used
memory block. Python would need a "compacting" garbage collection
to ensure that used memory is as tight as possible; then it could
free more aggressively. But, a "compacting" garbage collection would
make life **much** more difficult for extensions (implemented in C);
especially, thousands existing extensions would stop working.

Following the example of the C runtime library, Python manages free
memory internally. This implies that operating system means do not
show exact information about the amount of really used memory
(for the operation system, free memory in the internal memory
management is used).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Mike Dewhirst]

On 29/05/2020 2:59 pm, Dieter Maurer wrote:

Mike Dewhirst wrote at 2020-5-29 11:55 +1000:

On 29/05/2020 10:51 am, Terry Reedy wrote:
...

Source only releases only block Windows/Mac users who choose not to
upgrade to a released installer and who cannot or choose not to compile.

I am an example

I installed all the Pythons on my Windows 10 dev machine (locked into
Windows by having clients) but I'm also locked into Python 3.6.9 on my
Ubuntu 18.04 production machines.

You are not locked into Python 3.6 on Ubuntu: it is quite easy
to compile Python on *nix (this includes Ubuntu 18.04) yourself
(I did so recently for Python 3.9[a5]).


Thank you Dieter - yes I'm aware of that. You need to understand that I 
am lazy. I haven't done it before and I was persuaded by a grizzly old 
sysadmin that it is much easier to stick with the distro version.


I have actually compiled mod-wsgi before in the days of Python 2.7 so I 
do know how brilliantly Linux is set up for software development 
compared with Windows.


I will think about it seriously.

Cheers

Mike

--
https://mail.python.org/mailman/listinfo/python-list

Re: Is there some reason that recent Windows 3.6 releases don't included executable nor msi installers?

[Chris Angelico]

On Fri, May 29, 2020 at 3:15 PM Mike Dewhirst  wrote:
>
> On 29/05/2020 12:26 pm, Chris Angelico wrote:
> > On Fri, May 29, 2020 at 11:57 AM Mike Dewhirst  
> > wrote:
> >> I am an example
> >>
> >> I installed all the Pythons on my Windows 10 dev machine (locked into
> >> Windows by having clients) but I'm also locked into Python 3.6.9 on my
> >> Ubuntu 18.04 production machines.
> > Be careful of assuming too much here. The general policy with most
> > Linux distributions is to number the version according to the oldest
> > component in it, but they are free to backport whatever changes they
> > choose. I have Python 3.4.4 and 3.5.3 on this system (Debian Stretch),
> > but the 3.5 is numbered "3.5.3-1+deb9u1" which implies some collection
> > of additional patches. (I'd have to dig deep in the changelogs if I
> > cared exactly *which* patches.)
> >
> >> After chasing down an obscure problem I decided to go back to Py36 on
> >> Windows to be using the same versions in dev as in prd. I couldn't find
> >> an installer on python.org so I retrieved one (3.6.5) from my archives.
> > That'd be the same feature version, but depending exactly what the
> > problem is, there might not be *any* Windows build that exactly
> > corresponds.
> >
> >> If I was asked to suggest a guide for which versions ought to get a
> >> Windows binary I would look at the most popular LTS *nix distros and
> >> keep Windows binaries in step just to support people like me who cannot
> >> live with too much Windows clutter. Think of it as deeply humanitarian
> >> generosity.
> > Even if that were possible, who are you asking to do this? Whose time
> > is going to be put into finessing every point release to make sure
> > it's still buildable on Windows?
>
> I was careful to not ask anyone to do anything. "If I was asked ..."

You said that certain versions of Python "ought to get" Windows
binaries. They don't appear out of nowhere.

> The only reason I took up Python (after being burned by proprietary tool
> makers) was because it was promised to run on pretty much everything.
> Especially Apache platforms - which also run on Windows.

It does.

> It would be handy if Microsoft delivered Windows with Python installed
> but what can you expect from them if they can't see demand for it.

You can get Python from the Windows App Store or whatever they call it.

> There are lot of Python people working on Windows and they do so because
> they have no choice. Windows is simply the dominant platform. That's
> where most clients live.

And current versions of Python are easily available. It's just that
there's a limit to how many different versions people are willing to
go to the effort of building Windows binaries for.

You can easily get any GA release of Python for any supported
platform. Prerelease versions, old versions, or locally patched
versions, generally require that you build them yourself (regardless
of the platform).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ram memory not freed after executing python script on ubuntu system (rmlibre)

[Inada Naoki]

pymalloc manages only small blocks of memory.
Large (more than 512 byte) memory blocks are managed by malloc/free.

glibc malloc doesn't return much freed memory to OS.
You can try jemalloc instead of glibc.

On Ubuntu 20.04, you can try it by:

  LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so python your-script.py

On Fri, May 29, 2020 at 11:07 AM  wrote:
>
>
> We just ran into this problem when running our aiootp package's memory
> hard password hashing function (https://github.com/rmlibre/aiootp/). The
> memory was not being cleared after the function finished running but the
> script was still live. We tried making sure everything went out of scope
> and deleting things explicitly inside the function, but that didn't
> help. We even tried forcing the garbage collector to free up
> unreferenced memory with import gc; gc.collect(). But that only had a
> small dent in the memory that was being built up.
>
> The most useful answer online had to do with Python's free lists being
> created automatically when very large datasets were being processed
> (https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616).
>
>
> After putting the memory intensive work into a separate process, as the
> answer suggested, the memory was finally freed after execution.
>
> In our case, we wound up passing the result back in a
> ``multiprocessing.Manager().List()``. Though, passing your whole numpy
> dataset back doesn't seem feasible. I'd recommend doing the necessary
> memory intensive work in a separate process and passing only the
> necessary conclusions back to the main process.
>
> Or you could set up a multiprocessing queue to pass control messages to
> the spawned (daemon) process so it returns desired results on demand.
>
>
> Here's an excerpt from our code after the fix in case it's helpful.
>
> >@classmethod
> >def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256):
> >"""
> >The ``passcrypt`` function can be highly memory intensive.
> >These resources may not be freed up, & often are not, because of
> >python quirks around memory management. This is a huge problem.
> >So to force the release of those resources, we run the function
> >in another process which is guaranteed to release them.
> >"""
> >cls._validate_passcrypt_args(kb, cpu, hardness)
> >state = Manager().list()
> >process = Process(
> >target=cls._passcrypt,
> >args=(password, salt),
> >kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state),
> >)
> >process.start()
> >process.join()
> >return state.pop()
>
>
>
>
> On 2020-05-28 16:00, python-list-requ...@python.org wrote:
> > Send Python-list mailing list submissions to
> >   python-list@python.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >   https://mail.python.org/mailman/listinfo/python-list
> > or, via email, send a message with subject or body 'help' to
> >   python-list-requ...@python.org
> >
> > You can reach the person managing the list at
> >   python-list-ow...@python.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Python-list digest..."
> >
> > Today's Topics:
> >
> >1. Re: Custom logging function (zljubi...@gmail.com)
> >2. Ram memory not freed after executing python script on ubuntu
> >   system (Rahul Gupta)
> >3. Re: Ram memory not freed after executing python script on
> >   ubuntu system (Chris Angelico)
> >4. Re: Custom logging function (Peter Otten)
> >5. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >   (Peter Otten)
> >6. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >   (Chris Angelico)
> >7. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >   (Peter Otten)
> >8. Re: Ram memory not freed after executing python script on
> >   ubuntu system (Rahul Gupta)
> >9. Re: Ram memory not freed after executing python script on
> >   ubuntu system (Chris Angelico)
> >   10. Re: Behaviour of os.path.join (BlindAnagram)
> >   11. Constructing mime image attachment (Joseph L. Casale)
> >   12. Re: Behaviour of os.path.join (Eryk Sun)
> >   13. Re: Behaviour of os.path.join (Eryk Sun)
> >   14. Re: Behaviour of os.path.join (BlindAnagram)
> >   15. Re: Behaviour of os.path.join (Eryk Sun)
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Inada Naoki  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write tables from Word (.docx) to Excel (.xlsx) using xlsxwriter

[BBT]

On Thursday, 28 May 2020 03:07:48 UTC+8, Peter Otten  wrote:
> BBT wrote:
> 
> > I tried your code by replacing the Document portion:
> 
> > But I received an error:
> > TypeError: __init__() takes 1 positional argument but 2 were given
> 
> We seem to have different ideas of what replacing means. 
> Here is the suggested script spelt out:
> 
> import xlsxwriter
>  
> from docx.api import Document
> document = Document('/Users/xxx/Documents/xxx/Clauses Sample - Copy v1 - for 
> merge.docx')
> 
> wb = xlsxwriter.Workbook('C:/Users/xxx/Documents/xx/test clause 
> retrieval.xlsx')
> sheet = wb.add_worksheet("Compliance")
>  
> offset = 0
> for table in document.tables:
> for y, row in enumerate(table.rows):
> for x, cell in enumerate(row.cells):
> sheet.write(y + offset, x, cell.text)
> offset +=  len(table.rows) + 1  # one empty row between tables
> 
> wb.close()

Yes! That works well, thank you Peter! And I'm impressed by your 
object-oriented simulation too. Would look into it again :)
-- 
https://mail.python.org/mailman/listinfo/python-list