Re: Most "pythonic" syntax to use for an API client library

[Peter Otten]

Jonathan Leroy - Inikup via Python-list wrote:

> Hi all,
> 
> I'm writing a client library for a REST API. The API endpoints looks like
> this: /customers
> /customers/1
> /customers/1/update
> /customers/1/delete
> 
> Which of the following syntax do you expect an API client library to
> use, and why?
> 
> 1/
> api.customers_list()
> api.customers_info(1)
> api.customers_update(1, name='Bob')
> api.customers_delete(1)
> 
> 2/
> api.customers.list()
> api.customers.info(1)
> api.customers.update(1, name='Bob')
> api.customers.delete(1)
> 
> 3/
> api.customers.list()
> api.customers(1).info()
> api.customers(1).update(name='Bob')
> api.customers(1).delete()
> 
> ...any other?

How about mimicking (to some extent) an existing interface, like a list, 
dict, or set:

customers = api.customers

list(customers)  # __iter__

alice = customers[1]  # __getitem__

print(alice)  # __str__

alice.name = "Bob"  # __setattr__

del customers[42]  # __delitem__
del customers[alice]

> #3 seems to be more "pretty" to me, but I did not find any "official"
> recommendation online.
> 
> Thanks.
> 
> --
> Jonathan.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Most "pythonic" syntax to use for an API client library

[Thomas Jollans]

On 29/04/2019 09.18, Peter Otten wrote:
> Jonathan Leroy - Inikup via Python-list wrote:
> 
>> Hi all,
>>
>> I'm writing a client library for a REST API. The API endpoints looks like
>> this: /customers
>> /customers/1
>> /customers/1/update
>> /customers/1/delete
>>
>> Which of the following syntax do you expect an API client library to
>> use, and why?
>>
>> 1/
>> api.customers_list()
>> api.customers_info(1)
>> api.customers_update(1, name='Bob')
>> api.customers_delete(1)
>>
>> 2/
>> api.customers.list()
>> api.customers.info(1)
>> api.customers.update(1, name='Bob')
>> api.customers.delete(1)
>>
>> 3/
>> api.customers.list()
>> api.customers(1).info()
>> api.customers(1).update(name='Bob')
>> api.customers(1).delete()
>>
>> ...any other?
> 
> How about mimicking (to some extent) an existing interface, like a list, 
> dict, or set:
> 
> customers = api.customers
> 
> list(customers)  # __iter__
> 
> alice = customers[1]  # __getitem__
> 
> print(alice)  # __str__

This was my first thought seeing the third option as well, but...

> 
> alice.name = "Bob"  # __setattr__
> 
> del customers[42]  # __delitem__

do you want this sort of thing to update the upstream database directly?
Maybe there should be a commit() method on every object. Maybe not.

I imagine it would be best if the data is cached locally (i.e. alice =
customers[1] does a query, print(alice.name) does not). In this case you
probably want the local/database separation to be consistent for both
getting and setting things.

> del customers[alice]

Are you sure about this?

> 
>> #3 seems to be more "pretty" to me, but I did not find any "official"
>> recommendation online.

I'd like #3 if it had square brackets after .customers. Of the
suggestions as they are I prefer #2.

-- 
https://mail.python.org/mailman/listinfo/python-list

Python best practice instantiating classes in app

[Dave]

As apps get more complex we add modules, or Python files, to organize 
things.  One problem I have is a couple of data classes (list of 
dictionary objects) in a few modules that are used in a number of the 
other modules.  For example a list of meter reading dictionaries in one 
module is used by the user interface module to get the data from the 
user, a report module to display the data, and a file module to save and 
retrieve the data to/from file.  All the modules need to use the same 
instance of the list classes.


There are a number of ways to do this.  One is a module that creates the 
objects, then import that module into all of the others.  Works well, 
but may not be the best way to do the job.


A slight variation is to do this in the main module, but the main module 
has to be imported into the others.  Since I use the main module as a 
calling module to load data, start the user interface, and to close 
things down, it may not be the best place to also create the classes.


Another way to do this is have a line in each module to check the name. 
If it is the module name, then create the class and then import these 
modules in all the others.  A tad messy and maybe a little confusing.


So what are the suggestions from people that have been down this road 
before?


Thanks,
Dave
--
https://mail.python.org/mailman/listinfo/python-list

CheetahTemplate 3.2.2

[Oleg Broytman]

Hello!

I'm pleased to announce version 3.2.2, the second bugfix release of branch
3.2 of CheetahTemplate3.


What's new in CheetahTemplate3
==

Contributors for this release are
Pierre-Yves, Dan Vinakovsky, Nicolai Grodzitski.

Minor features:

  - Replaced outdated and insecure ``mktemp`` with ``mkstemp``.

Bug fixes:

  - Fixed bugs in ``TemplateCmdLineIface.py``: read binary pickles
from stdin and files.

Tests:

  - Use ``cgi.escape()`` for Python 2, ``html.escape()`` for Python 3.
  - Created tests for ``TemplateCmdLineIface``.


What is CheetahTemplate3


Cheetah3 is a free and open source template engine.
It's a fork of the original CheetahTemplate library.

Python 2.7 or 3.4+ is required.


Where is CheetahTemplate3
=

Site:
http://cheetahtemplate.org/

Development:
https://github.com/CheetahTemplate3

Download:
https://pypi.org/project/Cheetah3/3.2.2

News and changes:
http://cheetahtemplate.org/news.html

StackOverflow:
https://stackoverflow.com/questions/tagged/cheetah


Example
===

Below is a simple example of some Cheetah code, as you can see it's practically
Python. You can import, inherit and define methods just like in a regular Python
module, since that's what your Cheetah templates are compiled to :) ::

#from Cheetah.Template import Template
#extends Template

#set $people = [{'name' : 'Tom', 'mood' : 'Happy'}, {'name' : 'Dick',
'mood' : 'Sad'}, {'name' : 'Harry', 'mood' : 
'Hairy'}]

How are you feeling?

#for $person in $people

$person['name'] is $person['mood']

#end for


Oleg.
-- 
Oleg Broytmanhttps://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
-- 
https://mail.python.org/mailman/listinfo/python-list

Checking network input processing by Python for a multi-threaded server

[Markus Elfring]

Hello,

I constructed another multi-threaded TCP server for my needs
(based on the available software documentation).
https://docs.python.org/3/library/socketserver.html#asynchronous-mixins

I constructed also a corresponding script which should send nine record sets
(which get extracted from a simple JSON file) to this server on my local host
by the software “Python 3.7.2-3.1” (for an openSUSE system).
Related background information can be seen for a discussion topic like
“Data exchange over network interfaces by SmPL scripts”.
https://systeme.lip6.fr/pipermail/cocci/2019-April/005792.html
https://lore.kernel.org/cocci/6ec5b70f-39c3-79e5-608f-446a870f0...@web.de/

The file name for the client script is passed by a parameter to a command
which is repeated by this server in a loop.
It is evaluated then how often a known record set count was sent.
I got a test result like the following.


elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 
list_last_two_statements_from_if_branches-statistic-server3.py
"return code"|"received records"|"command output"|incidence
0|9||63
0|5||5
0|13||2
0|10||5
0|11||7
0|8||3
0|7||3
0|14||3
0|6||4
0|12||3
0|4||2

real1m23,301s
user0m6,434s
sys 0m1,430s


* How should this data processing approach be adjusted so that the same number
  of records will be handled in a consistent way?

* Which challenges from inter-process communication should be taken better into
  account for the involved programming interfaces (because of multi-threading)?

Regards,
Markus
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python best practice instantiating classes in app

[Terry Reedy]

On 4/29/2019 1:38 PM, Dave wrote:
As apps get more complex we add modules, or Python files, to organize 
things.  One problem I have is a couple of data classes (list of 
dictionary objects) in a few modules that are used in a number of the 
other modules.  For example a list of meter reading dictionaries in one 
module is used by the user interface module to get the data from the 
user, a report module to display the data, and a file module to save and 
retrieve the data to/from file.  All the modules need to use the same 
instance of the list classes.



There are a number of ways to do this.  One is a module that creates the 
objects, then import that module into all of the others.  Works well, 


You can have one *or more* such modules.  Perhaps you already do.


but may not be the best way to do the job.


In what way do you consider it unsatisfactory.



A slight variation is to do this in the main module, but the main module 
has to be imported into the others.


Avoid import loops unless really necessary.  They often work, but when 
they don't ... its a pain.


  Since I use the main module as a 
calling module to load data, start the user interface, and to close 
things down, it may not be the best place to also create the classes.


Another way to do this is have a line in each module to check the name. 
If it is the module name, then create the class and then import these 
modules in all the others.  A tad messy and maybe a little confusing.


Right.


--
Terry Jan Reedy


--
https://mail.python.org/mailman/listinfo/python-list

Re: Python best practice instantiating classes in app

[Dave]

On 4/29/19 3:26 PM, Terry Reedy wrote:

On 4/29/2019 1:38 PM, Dave wrote:
As apps get more complex we add modules, or Python files, to organize 
things.  One problem I have is a couple of data classes (list of 
dictionary objects) in a few modules that are used in a number of the 
other modules.  For example a list of meter reading dictionaries in 
one module is used by the user interface module to get the data from 
the user, a report module to display the data, and a file module to 
save and retrieve the data to/from file.  All the modules need to use 
the same instance of the list classes.



There are a number of ways to do this.  One is a module that creates 
the objects, then import that module into all of the others.  Works well, 


You can have one *or more* such modules.  Perhaps you already do.


but may not be the best way to do the job.


In what way do you consider it unsatisfactory.


It is not that I consider it unsatisfactory as much as I'm looking for 
the best way.  I am constantly amazed at the thought that has been given 
to this language (with the exception of access modifiers - perhaps, and 
lack of a case statement), and often find that there is a better way of 
doing everything.






A slight variation is to do this in the main module, but the main 
module has to be imported into the others.


Avoid import loops unless really necessary.  They often work, but when 
they don't ... its a pain.


  Since I use the main module as a calling module to load data, start 
the user interface, and to close things down, it may not be the best 
place to also create the classes.


Another way to do this is have a line in each module to check the 
name. If it is the module name, then create the class and then import 
these modules in all the others.  A tad messy and maybe a little 
confusing.


Right.




--
https://mail.python.org/mailman/listinfo/python-list

Generating generations of files

[DL Neil]

Are you aware of a library/utility which will generate and maintain the 
file names of multiple generations of a file?



The system generates multiple output files. For example, one might be 
called "output.rpt". However, we do not want to 'lose' the output 
file(s) from any previous run(s). In this case we want access to both 
files, eg "output.rpt" (the latest) and "output.rpt.1" (the previous 
generation, or maybe we could refer to it as a 'backup'). Similarly, if 
we run again, we want to maintain the three generations of the output 
file, including "output.rpt.2". (etc)


Backup systems/VCS often simply add a number to the original file-name, 
as above.


In logging, we only "rotate" the generations of a log file daily. Thus 
can add stat-data eg output.rpt.CCYY-MM-DD.


Yes, there are totally-unique naming ideas involving UUIDs. Um, "no"!

This application is multi-tenant (so 'your' output won't ever mix with 
'mine'). It is not multi-entrant/does not need to be 'thread-safe'. 
However, a single user might frequently make several runs in a day. 
Thus, leaning toward the 'add a number', rather than looking to extend 
the 'stat' idea down to hours and minutes.


OTOH, using generation-numbers when there are many versions, (?surely) 
requires a 'ripple' of renaming; whereas the date-time idea is 
one-time-only rename.


The users want the (latest) output files to continue to use their 
'traditional names'. The "generations' idea is (part of) this sprint's 
extension to the existing system. They have no particular preference for 
which/whatever generational naming system is used - as long as "it makes 
sense". They have undertaken responsibility for managing their disk-space(!)



Have looked in the PSL, eg os, os.path, shutil, pathlib... Hope I didn't 
miss something.


Any ideas (before we 'reinvent the wheel') please?
--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Chris Angelico]

On Tue, Apr 30, 2019 at 6:00 AM DL Neil  wrote:
>
> Are you aware of a library/utility which will generate and maintain the
> file names of multiple generations of a file?
>

Commit it to a git repository. All the generations have the same name,
but you can compare them, explore past versions, etc, etc, etc, with a
rich set of tools. And it's easy to invoke git from a program (if you
need information from stdout, most git commands accept a "--porcelain"
parameter), so you can fully automate.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Grant Edwards]

On 2019-04-29, DL Neil  wrote:

> Are you aware of a library/utility which will generate and maintain the 
> file names of multiple generations of a file?

Well, the FILES-11 filesystem on VAX/VMS did that automatically, but
that's probably not too helpful. Though I guess Python is actually
available for OpenVMS, and the porting of OpenVMS to AMD64 (aka
x86-64) is progressing: https://www.vmssoftware.com/updates_port.html

-- 
Grant Edwards   grant.b.edwardsYow! They collapsed
  at   ... like nuns in the
  gmail.comstreet ... they had no
   teen appeal!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[MRAB]

On 2019-04-29 20:59, DL Neil wrote:

Are you aware of a library/utility which will generate and maintain the
file names of multiple generations of a file?


The system generates multiple output files. For example, one might be
called "output.rpt". However, we do not want to 'lose' the output
file(s) from any previous run(s). In this case we want access to both
files, eg "output.rpt" (the latest) and "output.rpt.1" (the previous
generation, or maybe we could refer to it as a 'backup'). Similarly, if
we run again, we want to maintain the three generations of the output
file, including "output.rpt.2". (etc)

Backup systems/VCS often simply add a number to the original file-name,
as above.

In logging, we only "rotate" the generations of a log file daily. Thus
can add stat-data eg output.rpt.CCYY-MM-DD.

Yes, there are totally-unique naming ideas involving UUIDs. Um, "no"!

This application is multi-tenant (so 'your' output won't ever mix with
'mine'). It is not multi-entrant/does not need to be 'thread-safe'.
However, a single user might frequently make several runs in a day.
Thus, leaning toward the 'add a number', rather than looking to extend
the 'stat' idea down to hours and minutes.

OTOH, using generation-numbers when there are many versions, (?surely)
requires a 'ripple' of renaming; whereas the date-time idea is
one-time-only rename.

The users want the (latest) output files to continue to use their
'traditional names'. The "generations' idea is (part of) this sprint's
extension to the existing system. They have no particular preference for
which/whatever generational naming system is used - as long as "it makes
sense". They have undertaken responsibility for managing their disk-space(!)


Have looked in the PSL, eg os, os.path, shutil, pathlib... Hope I didn't
miss something.

Any ideas (before we 'reinvent the wheel') please?


Why would generation numbers result in a 'ripple' of renaming?

You're assuming that "output.rpt.1" comes after "output.rpt.2", but it 
could just as well come before (generation 1 precedes generation 2, 
etc.). You're just left with the exception of the unnumbered 
"output.rpt" being the latest.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Peter J. Holzer]

On 2019-04-29 20:12:28 -, Grant Edwards wrote:
> On 2019-04-29, DL Neil  wrote:
> > Are you aware of a library/utility which will generate and maintain the 
> > file names of multiple generations of a file?
> 
> Well, the FILES-11 filesystem on VAX/VMS did that automatically, but
> that's probably not too helpful. Though I guess Python is actually
> available for OpenVMS, and the porting of OpenVMS to AMD64 (aka
> x86-64) is progressing: https://www.vmssoftware.com/updates_port.html

Until this is finished you could use something like this:

#!/usr/bin/python3

import os

def open11(file, mode, **kwargs):
if "w" in mode:
try:
oldfile = os.readlink(file)
basename, version = oldfile.split(";")
except FileNotFoundError:
basename = os.path.basename(file)
version = 0
newfile = basename + ";" + str(int(version) + 1)
os.unlink(file)
os.symlink(newfile, file)
return open(file, mode, **kwargs)


if __name__ == "__main__":
with open11("foo", "w") as f:
f.write("test1")

with open11("foo", "w") as f:
f.write("test2")

with open11("foo", "w") as f:
f.write("test3")

:-)

(WARNING: I haven't really tested this)

hp

PS: I like Chris' suggestion to just use git. But it's IMHO only
practical if you plan to keep old versions for ever.

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Get the source of a class reliably?!?

[computermaster360 .]

Does anyone have an idea why classes don't contain their definition
line number as functions or methods do?

>>> some_fun.__code__.co_firstlineno
123
>>> SomeClass.???

This leads to some funny stuff when using `inspect`, such as this:

-- weird.py -
"""
class C:
  HAHAHA! YOU FOOL!
"""

class C:
  "this is a perfectly ok class"

  class C:
"this class is nice"
-

>>> inspect.getsource(weird.C)
class C:
  HAHAHA! YOU FOOL!


Why ???
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Eli the Bearded]

In comp.lang.python, Peter J. Holzer  wrote:
> On 2019-04-29 20:12:28 -, Grant Edwards wrote:
>> Well, the FILES-11 filesystem on VAX/VMS did that automatically, but
>> that's probably not too helpful.
> Until this is finished you could use something like this:
> 
> #!/usr/bin/python3
> 
> import os
> 
> def open11(file, mode, **kwargs):
> if "w" in mode:
> try:
> oldfile = os.readlink(file)
> basename, version = oldfile.split(";")
> except FileNotFoundError:
> basename = os.path.basename(file)
> version = 0
> newfile = basename + ";" + str(int(version) + 1)
> os.unlink(file)
> os.symlink(newfile, file)
> return open(file, mode, **kwargs)
> 
> 
> if __name__ == "__main__":
> with open11("foo", "w") as f:
> f.write("test1")
> 
> with open11("foo", "w") as f:
> f.write("test2")
> 
> with open11("foo", "w") as f:
> f.write("test3")
> 
> :-)

Noted.

> (WARNING: I haven't really tested this)

No foo:

Traceback (most recent call last):
  File "versioned-open", line 21, in 
with open11("foo", "w") as f:
  File "versioned-open", line 15, in open11
os.unlink(file)
FileNotFoundError: [Errno 2] No such file or directory: 'foo'

There is a foo, but it's not a symlink:

Traceback (most recent call last):
  File "versioned-open", line 21, in 
with open11("foo", "w") as f:
  File "versioned-open", line 9, in open11
oldfile = os.readlink(file)
OSError: [Errno 22] Invalid argument: 'foo'

Parse error on version string:

Traceback (most recent call last):
  File "versioned-open", line 21, in 
with open11("foo", "w") as f:
  File "versioned-open", line 10, in open11
basename, version = oldfile.split(";")
ValueError: not enough values to unpack (expected 2, got 1)

etc (there are several possible parse errors: no semicolon, multiple
semicolons, invalid literal for int()).

That said, I do think it makes a reasonable suggestion for the stated
versioning requirement.

Elijah
--
bet a FAT filesystem would produce a different error
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Chris Angelico]

On Tue, Apr 30, 2019 at 8:16 AM Eli the Bearded <*@eli.users.panix.com> wrote:
> bet a FAT filesystem would produce a different error

Probably it'd raise BadFileSystemError or something. Which is a
subclass of OSError, SystemError, TimeoutError, OverflowError,
BlockingIOError, and SystemExit.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 8:12 AM, Grant Edwards wrote:

On 2019-04-29, DL Neil  wrote:


Are you aware of a library/utility which will generate and maintain the
file names of multiple generations of a file?


Well, the FILES-11 filesystem on VAX/VMS did that automatically, but
that's probably not too helpful. Though I guess Python is actually
available for OpenVMS, and the porting of OpenVMS to AMD64 (aka
x86-64) is progressing: https://www.vmssoftware.com/updates_port.html



Wow = blast from the past!

Hopefully, not the only place from where I remember such! At various 
times, people have wondered how I was able to very quickly pick-up the 
commandLN of a 'new' OpSys, eg CP/M, PC-/MS-DOS, Linux. They are all so 
similar to RT-11 and RSTS (PDP-11 line), and thus VAX VMS!


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 8:17 AM, MRAB wrote:

On 2019-04-29 20:59, DL Neil wrote:

Are you aware of a library/utility which will generate and maintain the
file names of multiple generations of a file?


OTOH, using generation-numbers when there are many versions, (?surely)
requires a 'ripple' of renaming; whereas the date-time idea is
one-time-only rename.




Why would generation numbers result in a 'ripple' of renaming?

You're assuming that "output.rpt.1" comes after "output.rpt.2", but it 
could just as well come before (generation 1 precedes generation 2, 
etc.). You're just left with the exception of the unnumbered 
"output.rpt" being the latest.



I was!

However, that's the way statisticians and mathematicians think, and thus 
expressed themselves (I 'swapped' the minus-sign for a fileNM separator):


(current) version   version.rpt
-1 version  version.rpt.1
-2 version  version.rpt.2
etc

Thank you for exposing my assumption...
(guilty grin)

How would you rate my chances of talking them into the idea?
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 9:04 AM, Peter J. Holzer wrote:

On 2019-04-29 20:12:28 -, Grant Edwards wrote:

On 2019-04-29, DL Neil  wrote:

Are you aware of a library/utility which will generate and maintain the
file names of multiple generations of a file?


Well, the FILES-11 filesystem on VAX/VMS did that automatically, but
that's probably not too helpful. Though I guess Python is actually
available for OpenVMS, and the porting of OpenVMS to AMD64 (aka
x86-64) is progressing: https://www.vmssoftware.com/updates_port.html


Until this is finished you could use something like this:

#!/usr/bin/python3

import os

def open11(file, mode, **kwargs):
 if "w" in mode:
 try:
 oldfile = os.readlink(file)
 basename, version = oldfile.split(";")
 except FileNotFoundError:
 basename = os.path.basename(file)
 version = 0
 newfile = basename + ";" + str(int(version) + 1)
 os.unlink(file)
 os.symlink(newfile, file)
 return open(file, mode, **kwargs)


if __name__ == "__main__":
 with open11("foo", "w") as f:
 f.write("test1")

 with open11("foo", "w") as f:
 f.write("test2")

 with open11("foo", "w") as f:
 f.write("test3")

:-)

(WARNING: I haven't really tested this)



Didn't think of the concept of using links!

As Eli showed, with a bit of refinement, we can make something from here...

Thanks!
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 10:59 AM, Chris Angelico wrote:

On Tue, Apr 30, 2019 at 8:16 AM Eli the Bearded <*@eli.users.panix.com> wrote:

bet a FAT filesystem would produce a different error


Probably it'd raise BadFileSystemError or something. Which is a
subclass of OSError, SystemError, TimeoutError, OverflowError,
BlockingIOError, and SystemExit.



Fortunately, it runs on a Linux 'compute server'.

(although, if they're going to keep-and-keep stuff, perhaps I'd better 
re-spec to a more appropriate file storage m/c... Thanks!)

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Eli the Bearded]

In comp.lang.python, DL Neil   wrote:
> On 30/04/19 10:59 AM, Chris Angelico wrote:
>>> bet a FAT filesystem would produce a different error
>> Probably it'd raise BadFileSystemError or something. Which is a
> Fortunately, it runs on a Linux 'compute server'.

I mount FAT under Linux all the time.

Elijah
--
but generally for sneakernet reasons
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 8:04 AM, Chris Angelico wrote:

On Tue, Apr 30, 2019 at 6:00 AM DL Neil  wrote:


Are you aware of a library/utility which will generate and maintain the
file names of multiple generations of a file?



Commit it to a git repository. All the generations have the same name,
but you can compare them, explore past versions, etc, etc, etc, with a
rich set of tools. And it's easy to invoke git from a program (if you
need information from stdout, most git commands accept a "--porcelain"
parameter), so you can fully automate.



Great idea!
(and all 'the work' is already done)

I've seen various Py + GIT entries in the library, so can I assume the 
ability to output a file directly into a GIT repo/branch, instead of the 
file-system?


Is this a common use-case?


Three immediate thoughts:

1 I'd have to put GIT onto one of the users' storage servers (see 
comment elsewhere about re-thinking location of file-storage). OK, so 
I'm lazy - but I work hard at it!


2 We'd need a user-friendly GIT-client for the users. The words "Linus" 
and "user friendly" don't often sit together happily in the same 
sentence (and if he ever reads this...)


3 The users' case for retaining older-versions is so that they can 
compare between runs. (I blame myself, showing them filecmp, noting that 
they could quickly employ Calc/Excel...)
Easiest explained example = some slight change in a variable results in 
a 'surprise' result. (which scenario is almost identical to the 
'sensation' when a user throws some data at *our code* and finds some 
'error' - schadenfreude means there's always a silver lining!)
Earlier, I had suggested dropping the o/p of each 'run' into its own 
sub-dir of the client-/experiment-directory (perhaps using date-time) 
and thereby completely avoiding any fileNM 'collisions'. Lead balloon! 
Sigh...



Thanks! I'll have some little git - apologies, I mean: "a Git expert", 
research further...

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Chris Angelico]

On Tue, Apr 30, 2019 at 9:46 AM Eli the Bearded <*@eli.users.panix.com> wrote:
>
> In comp.lang.python, DL Neil   wrote:
> > On 30/04/19 10:59 AM, Chris Angelico wrote:
> >>> bet a FAT filesystem would produce a different error
> >> Probably it'd raise BadFileSystemError or something. Which is a
> > Fortunately, it runs on a Linux 'compute server'.
>
> I mount FAT under Linux all the time.
>
> but generally for sneakernet reasons

Hmm, really? Even for compatibility with Windows, I haven't used FAT
in years - normally a USB stick will be formatted NTFS. But, whatever.
Can use whatever you like!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Chris Angelico]

On Tue, Apr 30, 2019 at 9:54 AM DL Neil  wrote:
>
> On 30/04/19 8:04 AM, Chris Angelico wrote:
> > On Tue, Apr 30, 2019 at 6:00 AM DL Neil  
> > wrote:
> >>
> >> Are you aware of a library/utility which will generate and maintain the
> >> file names of multiple generations of a file?
> >>
> >
> > Commit it to a git repository. All the generations have the same name,
> > but you can compare them, explore past versions, etc, etc, etc, with a
> > rich set of tools. And it's easy to invoke git from a program (if you
> > need information from stdout, most git commands accept a "--porcelain"
> > parameter), so you can fully automate.
>
>
> Great idea!
> (and all 'the work' is already done)
>
> I've seen various Py + GIT entries in the library, so can I assume the
> ability to output a file directly into a GIT repo/branch, instead of the
> file-system?
>
> Is this a common use-case?

Common? Not sure. Possible? Absolutely! This is a script that creates
a commit on a branch other than the currently checked-out one:

https://github.com/Rosuav/shed/blob/master/git-deploy#L48

Most of the time, you'll just edit the file and then commit, since
you'll generally want to have the most current version visible in the
file system. But you can create the new version first, and then
separately "deploy" that version by moving the branch pointer.

> Three immediate thoughts:
>
> 1 I'd have to put GIT onto one of the users' storage servers (see
> comment elsewhere about re-thinking location of file-storage). OK, so
> I'm lazy - but I work hard at it!

Fair point. Not too hard though.

> 2 We'd need a user-friendly GIT-client for the users. The words "Linus"
> and "user friendly" don't often sit together happily in the same
> sentence (and if he ever reads this...)

There are plenty of user-friendly git clients that do "normal"
operations (for various definitions of "normal", but generally that
includes committing changes on the file system to the current branch).
If your use-case differs, you can easily create your own UI that calls
on lower-level git tools (such as the script linked above - you can
run that as if "git deploy" were a core git subcommand).

> 3 The users' case for retaining older-versions is so that they can
> compare between runs. (I blame myself, showing them filecmp, noting that
> they could quickly employ Calc/Excel...)
> Easiest explained example = some slight change in a variable results in
> a 'surprise' result. (which scenario is almost identical to the
> 'sensation' when a user throws some data at *our code* and finds some
> 'error' - schadenfreude means there's always a silver lining!)

Depending on exactly how this works, it might be easiest to just
commit the "current state" (for some definition of "current"), and
then just edit the file directly - "git diff" will tell you what's
changed since commit. That's how I track changes to other people's
files; for instance, whenever Counter-Strike: Global Offensive has an
update, I check its changes and then create a commit to accept those
changes.

> Earlier, I had suggested dropping the o/p of each 'run' into its own
> sub-dir of the client-/experiment-directory (perhaps using date-time)
> and thereby completely avoiding any fileNM 'collisions'. Lead balloon!
> Sigh...

Okay, so this needs to be as smooth and invisible as possible. If you
don't need any concurrency, the easiest way might be for each run to
automatically do a "git commit -a" at the end, and then you can
compare the most recent run to the run prior to that with "git show".

I would recommend planning out a workflow on the assumption that git
is magically able to do whatever you want it to, and then try to
figure out how to make git do it. Chances are you can do most of it
with simple high-level commands (perhaps automatically executed), and
then if there's something you can't do that way, it might be time to
delve into the lower level tools. It's almost certainly possible.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[DL Neil]

On 30/04/19 11:57 AM, Chris Angelico wrote:

On Tue, Apr 30, 2019 at 9:46 AM Eli the Bearded <*@eli.users.panix.com> wrote:


In comp.lang.python, DL Neil   wrote:

On 30/04/19 10:59 AM, Chris Angelico wrote:

bet a FAT filesystem would produce a different error

Probably it'd raise BadFileSystemError or something. Which is a

Fortunately, it runs on a Linux 'compute server'.


I mount FAT under Linux all the time.


+1



but generally for sneakernet reasons


Hmm, really? Even for compatibility with Windows, I haven't used FAT
in years - normally a USB stick will be formatted NTFS. But, whatever.
Can use whatever you like!



Wasn't his real error the word "under"?
(and having just observed our Git specialist's inexorably-expanding 
waistline)

Doesn't the FAT surround the lean?

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Cameron Simpson]

On 30Apr2019 11:24, DL Neil  wrote:

On 30/04/19 8:17 AM, MRAB wrote:

Why would generation numbers result in a 'ripple' of renaming?

You're assuming that "output.rpt.1" comes after "output.rpt.2", but 
it could just as well come before (generation 1 precedes generation 
2, etc.). You're just left with the exception of the unnumbered 
"output.rpt" being the latest.


I was!

However, that's the way statisticians and mathematicians think, and 
thus expressed themselves (I 'swapped' the minus-sign for a fileNM 
separator):


(current) version   version.rpt
-1 version  version.rpt.1
-2 version  version.rpt.2
etc


For a counter example, I'm pretty sure the VMS built in file versioning 
went on the scheme MRAB described: rewriting version.rpt caused the old 
version to become "version.rpt;n" where n counted up from 1.


Of course this is an arbitrary convention.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Generating generations of files

[Gene Heskett]

On Monday 29 April 2019 20:20:50 DL Neil wrote:

> On 30/04/19 11:57 AM, Chris Angelico wrote:
> > On Tue, Apr 30, 2019 at 9:46 AM Eli the Bearded 
<*@eli.users.panix.com> wrote:
> >> In comp.lang.python, DL Neil   
wrote:
> >>> On 30/04/19 10:59 AM, Chris Angelico wrote:
> > bet a FAT filesystem would produce a different error
> 
>  Probably it'd raise BadFileSystemError or something. Which is a
> >>>
> >>> Fortunately, it runs on a Linux 'compute server'.
> >>
> >> I mount FAT under Linux all the time.
> +1
> +1 
> >> but generally for sneakernet reasons
> >
> > Hmm, really? Even for compatibility with Windows, I haven't used FAT
> > in years - normally a USB stick will be formatted NTFS. But,
> > whatever. Can use whatever you like!
>
> Wasn't his real error the word "under"?
> (and having just observed our Git specialist's inexorably-expanding
> waistline)
> Doesn't the FAT surround the lean?
>
> --
> Regards =dn


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python best practice instantiating classes in app

[DL Neil]

Dave,


On 30/04/19 7:36 AM, Dave wrote:

On 4/29/19 3:26 PM, Terry Reedy wrote:

On 4/29/2019 1:38 PM, Dave wrote:
As apps get more complex we add modules, or Python files, to organize 
things.  One problem I have is a couple of data classes (list of 
dictionary objects) in a few modules that are used in a number of the 
other modules.  For example a list of meter reading dictionaries in 
one module is used by the user interface module to get the data from 
the user, a report module to display the data, and a file module to 
save and retrieve the data to/from file.  All the modules need to use 
the same instance of the list classes.



There are a number of ways to do this.  One is a module that creates 
the objects, then import that module into all of the others.  Works 
well, 


You can have one *or more* such modules.  Perhaps you already do.


but may not be the best way to do the job.


In what way do you consider it unsatisfactory.


It is not that I consider it unsatisfactory as much as I'm looking for 
the best way.  I am constantly amazed at the thought that has been given 
to this language (with the exception of access modifiers - perhaps, and 
lack of a case statement), and often find that there is a better way of 
doing everything.






A slight variation is to do this in the main module, but the main 
module has to be imported into the others.


Avoid import loops unless really necessary.  They often work, but when 
they don't ... its a pain.


  Since I use the main module as a calling module to load data, start 
the user interface, and to close things down, it may not be the best 
place to also create the classes.


Another way to do this is have a line in each module to check the 
name. If it is the module name, then create the class and then import 
these modules in all the others.  A tad messy and maybe a little 
confusing.


Right.


My preference is for as little 'clutter' on the 'mainline' as possible. 
Reading the code (and comments) should illustrate the approach being 
taken and provide an overview. NB Others may prefer otherwise, eg many 
PSL packages!


That means most/all classes are import-ed from a package/module.

So, the 'mainline' features:

from meters import meter_dict
from meters import Meter# class named Meter
...

meter = Meter( type=meter_dict[ key ] )


This allows any/all related applications to do the same.

Again, IMHO, import-ed modules are best kept-organised in a separate 
directory.


As long as all of these 'related applications' which will import the 
module can be kept in one dir, then the modules can be located in a 
sub-dir, ie a sub-dir common to all apps.


However, if there is a need to separate applications, eg security/access 
control, then life becomes a little more complex...


The other issue is that if two new programs are designed together and 
import the same module, everything will work well - at the start. 
However, if the module must be changed to suit (only) one of the 
applications, then such changes need to be made with (all) the other 
applications in-mind!


Failing that, you'll wind-up with multiple versions of modules. Here be 
dragons!


There was a discussion 'here' a few weeks ago, which may yield some 
interesting thoughts: "Handy utilities = Friday Filosofical Finking". 
Check in the archive (access link, as footer)...

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Checking network input processing by Python for a multi-threaded server

[dieter]

Markus Elfring  writes:
> ...
> I constructed another multi-threaded TCP server for my needs
> (based on the available software documentation).
> https://docs.python.org/3/library/socketserver.html#asynchronous-mixins
> ...
> elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 
> list_last_two_statements_from_if_branches-statistic-server3.py
> "return code"|"received records"|"command output"|incidence
> 0|9||63
> 0|5||5
> 0|13||2
> 0|10||5
> 0|11||7
> 0|8||3
> 0|7||3
> 0|14||3
> 0|6||4
> 0|12||3
> 0|4||2
>
> real  1m23,301s
> user  0m6,434s
> sys   0m1,430s
>
>
> * How should this data processing approach be adjusted so that the same number
>   of records will be handled in a consistent way?

Does this mean that the second number in a listing like the above
should be approximatively equal?

Then, obviously, your threads must communicate among themselves
and wait in case they have too much run ahead.

Python by itself does not garantee a fair scheduling of threads
(and of course knows nothing about the number of processed records).
It runs a thread for a number of instructions or until it releases
the so called "GIL" (= "Global Interpreter Lock") volontarily (which
it typically does when it waits or starts a typically "longer" low
level operation (such as reading from a file)). Then it
gives another thread a chance. When I remember right, then
the underlying C library decides which other thread starts.
This would mean that the level of fairness is determined by
the C library.


> * Which challenges from inter-process communication should be taken better 
> into
>   account for the involved programming interfaces (because of 
> multi-threading)?
Multi-threading is "inner-process" not "inter-process".

As you say to have created a "multi-threaded TCP server", I assume
that your server spawns a thread for each TCP connection.
Then, you can concentrate on proper multi-thread synchronization
and forget about "inter-process communication challenges".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Get the source of a class reliably?!?

[dieter]

"computermaster360 ."  writes:

> Does anyone have an idea why classes don't contain their definition
> line number as functions or methods do?
>
 some_fun.__code__.co_firstlineno
> 123

Because classes do not have associated "code" objects.

As you see above, it is not the function itself ("some_fun" in
your case) that carries the line information but the associated
"code object" ("some_fun.__code__").

Functions typically are executed several times and
each execution needs to execute the associated code;
therefore, functions reference this code.
There is no specific code directly associated with a class: 
general (i.e. not class specific) code is used to collect
the operands for the class contruction (i.e. base classes, attribute/method
definitions), then the class' metaclass is used to really construct
the class object. Therefore, class objects have no associated "code object".


> ...
> This leads to some funny stuff when using `inspect`, such as this:
>
> -- weird.py -
> """
> class C:
>   HAHAHA! YOU FOOL!
> """
>
> class C:
>   "this is a perfectly ok class"
> ...
 inspect.getsource(weird.C)
> class C:
>   HAHAHA! YOU FOOL!
>
>
> Why ???

In order to answer questions of this kind yourself, you
can use the fact that Python is open source.

A look into "inspect.py" shows that Python uses the following
code to find the source lines for a class:

if isclass(object):
name = object.__name__
pat = re.compile(r'^(\s*)class\s*' + name + r'\b')
# make some effort to find the best matching class definition:
# use the one with the least indentation, which is the one
# that's most probably not inside a function definition.
candidates = []
for i in range(len(lines)):
match = pat.match(lines[i])
if match:
# if it's at toplevel, it's already the best one
if lines[i][0] == 'c':
return lines, i
# else add whitespace to candidate list
candidates.append((match.group(1), i))
if candidates:
# this will sort by whitespace, and by line number,
# less whitespace first
candidates.sort()
return lines, candidates[0][1]

The comment "make some effort ..." indicates that the author
knew that the result will not be correct in all cases.
You have found such a case.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Get the source of a class reliably?!?

[Chris Angelico]

On Tue, Apr 30, 2019 at 4:40 PM dieter  wrote:
>
> "computermaster360 ."  writes:
>
> > Does anyone have an idea why classes don't contain their definition
> > line number as functions or methods do?
> >
>  some_fun.__code__.co_firstlineno
> > 123
>
> Because classes do not have associated "code" objects.
>
> As you see above, it is not the function itself ("some_fun" in
> your case) that carries the line information but the associated
> "code object" ("some_fun.__code__").
>
> Functions typically are executed several times and
> each execution needs to execute the associated code;
> therefore, functions reference this code.
> There is no specific code directly associated with a class:
> general (i.e. not class specific) code is used to collect
> the operands for the class contruction (i.e. base classes, attribute/method
> definitions), then the class' metaclass is used to really construct
> the class object. Therefore, class objects have no associated "code object".
>

I'm not sure if it'd be of any value to the OP, but a class *does*
have a code object associated with it; it's just that it isn't
retained by the class object after execution. You can see it by
looking at a function that creates a class:

>>> def f():
... class C:
... """Attempt 1"""
... class C:
... """Attempt 2"""
... return C
...
>>> f.__code__.co_consts[1]
", line 2>
>>> f.__code__.co_consts[3]
", line 4>

(If you try this yourself, I would recommend just looking at all of
co_consts, as the exact indices are immaterial.)

I'm not aware of a way to get from a class to the code object that
created it, though, any more than you can (in any general way) get
from a function's return value back to the function that created it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list