Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[boB Stepp]

On Sun, Oct 21, 2018 at 1:47 AM Quentin Agren  wrote:
>
> Hi Robert,
>
> Far from being an expert, my two cents on this:
>
> - As I understand it, you should at least use the 'dir' parameter to 
> NamedTemporaryFile,  otherwise your files will be created in a '/tmp/'-like 
> directory that may be wiped clean by the OS now and then.

I am planning to set both the "dir" and "suffix" parameters if I go
this route and probably the "mode" parameter, too.

> - I seems that the only functionality you desire from tempfile is the 
> generation of a random file name (and maybe ensuring that no collision 
> occurs). You could use the 'random' standard library module to generate such 
> names easily (which is about what tempfile does under the hood)

It is important for me to have no name collisions however unlikely
such an event might be.

> import random
> CHARS = 'abcdefghijklmnopqrstuvw1234567890'
> def random_name(length):
> return ''.join([random.choice(CHARS) for _ in range(length)])

It occurred to me to do this type of approach.  But why write this
myself when a standard library module may give me what I want with
only one or two lines of code?  When I searched online others have
recommended using the uuid library to generate names.  But one still
has to check that it does not already exist (However unlikely.) and
format the final filename.  And I would have to do the same for the
code snippet you supplied, adding a few additional lines of code.

But thank you for your input!  It may turn out that there is something
undesirable that I am unaware of in the NamedTemporaryFile approach,
other than what I really want is a NamedPermanentFile approach, but
the standard library naming suggest otherwise! ~(:>))

boB
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] organisation of folder structure of Python's installation directory

[Malhar Lathkar]

Hi all.

I am trying to understand how Python's installation directory is organized.
I find lib as well as libs folder. what's the difference? I couldn't find
exact explanation of purpose of various other sub folders like include,
scripts, share etc. Can you help me understand this? Please share
documentation link if any.

thanks.

-- 
Malhar Lathkar


 [image: in.linkedin.com/in/malharlathkar]


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Some questions about importlib

[Quentin Agren]

Hi,

My name Quentin, and this is my first post to this list so please redirect
me if this is not the proper audience.

I have been studying the 'importlib' standard library package this past
week, and although I find it very readable I am puzzled by some questions
that I'd like to share.

- [Chicken and egg] Does the 'import' statement in Python make use of
'importlib'? If so, how are the imports in 'importlib' itself carried out?
(for example in __init__.py)

- Why does 'importlib' use 'nt' or 'psoix' (see
'_bootstrap_external._setup()') rather than the portable 'os', and
reimplement some of the latter's functionality, for example '_path_join',
or '_path_split'. Another example is in 'SourceFileLoader.set_data()' where
the logic of 'os.mkdirs' is reproduced to create all necessary intermediary
directories in a file path.

- Similarly, would not using the 'struct' module simplify the
packing/unpacking of bytecode. For example by defining BYTECODE_HEADER_FMT
= '4sII' (Python 3.6)

Thanks for any insights you may share and have a nice sunday,
Quentin
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[Quentin Agren]

Hi Robert,

Far from being an expert, my two cents on this:

- As I understand it, you should at least use the 'dir' parameter to
NamedTemporaryFile,  otherwise your files will be created in a '/tmp/'-like
directory that may be wiped clean by the OS now and then.
- I seems that the only functionality you desire from tempfile is the
generation of a random file name (and maybe ensuring that no collision
occurs). You could use the 'random' standard library module to generate
such names easily (which is about what tempfile does under the hood)

import random
CHARS = 'abcdefghijklmnopqrstuvw1234567890'
def random_name(length):
return ''.join([random.choice(CHARS) for _ in range(length)])

Cheers,
Quentin

Le dim. 21 oct. 2018 à 08:15, boB Stepp  a écrit :

> Use case:  I want to allow a user of my Solitaire Scorekeeper program
> to be able to give any name he wants to each game of solitaire he
> wishes to record.  My thought for permanent storage of each game's
> parameters is to use a dictionary to map the user-chosen game names to
> a unique json filename.  This way I hope no shenanigans can occur with
> weird characters/non-printing characters.
>
> My initial thought was to just have a sequence of game names with
> incrementing numerical suffixes:  game_0, game_1, ... , game_n.  But
> this would require the program to keep track of what the next
> available numerical suffix is.  Additionally, if a user chooses to
> delete a game, then there would be a gap in the numerical sequence of
> the game filenames.  I find such a gap aesthetically displeasing and
> would probably go to additional efforts to reuse such deleted
> filenames, so there would be no such "gaps".
>
> So I am now wondering if using
> tempfile.NamedTemporaryFile(delete=False) would solve this problem
> nicely?  As I am not very familiar with this library, are there any
> unforeseen issues I should be made aware of?  Would this work equally
> well on all operating systems?
>
> TIA!
> --
> boB
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[Alan Gauld via Tutor]

On 21/10/18 07:13, boB Stepp wrote:

> My initial thought was to just have a sequence of game names with
> incrementing numerical suffixes:  game_0, game_1, ... , game_n.  But
> this would require the program to keep track of what the next
> available numerical suffix is.

The traditional approach to such scenarios (eg. for log files)
is to append the datetime. If that is not enough granularity
you can additionally append a numeric suffix that applies
only during a single session(ie a global variable) and is
only applied when the datetimes are identical - very rare.

> tempfile.NamedTemporaryFile(delete=False) would solve this problem

Can't help there, I've never used it. :-)

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[Cameron Simpson]

On 21Oct2018 01:13, boB Stepp  wrote:

Use case:  I want to allow a user of my Solitaire Scorekeeper program
to be able to give any name he wants to each game of solitaire he
wishes to record.  My thought for permanent storage of each game's
parameters is to use a dictionary to map the user-chosen game names to
a unique json filename.  This way I hope no shenanigans can occur with
weird characters/non-printing characters.

My initial thought was to just have a sequence of game names with
incrementing numerical suffixes:  game_0, game_1, ... , game_n.  But
this would require the program to keep track of what the next
available numerical suffix is.  Additionally, if a user chooses to
delete a game, then there would be a gap in the numerical sequence of
the game filenames.  I find such a gap aesthetically displeasing and
would probably go to additional efforts to reuse such deleted
filenames, so there would be no such "gaps".

So I am now wondering if using
tempfile.NamedTemporaryFile(delete=False) would solve this problem
nicely?  As I am not very familiar with this library, are there any
unforeseen issues I should be made aware of?  Would this work equally
well on all operating systems?


The doco reads that way to me.

However, NamedTemporaryFile is a (nice) wrapper for tempfile.mkstemp().  
Why not use that directly?


Cheers,
Cameron Simpson 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[Peter Otten]

boB Stepp wrote:

> Use case:  I want to allow a user of my Solitaire Scorekeeper program
> to be able to give any name he wants to each game of solitaire he
> wishes to record.  My thought for permanent storage of each game's
> parameters is to use a dictionary to map the user-chosen game names to
> a unique json filename.  This way I hope no shenanigans can occur with
> weird characters/non-printing characters.
> 
> My initial thought was to just have a sequence of game names with
> incrementing numerical suffixes:  game_0, game_1, ... , game_n.  But
> this would require the program to keep track of what the next
> available numerical suffix is.  Additionally, if a user chooses to
> delete a game, then there would be a gap in the numerical sequence of
> the game filenames.  I find such a gap aesthetically displeasing and
> would probably go to additional efforts to reuse such deleted
> filenames, so there would be no such "gaps".
> 
> So I am now wondering if using
> tempfile.NamedTemporaryFile(delete=False) would solve this problem
> nicely?  As I am not very familiar with this library, are there any
> unforeseen issues I should be made aware of?  Would this work equally
> well on all operating systems?

I think this is cool thinking outside of the box. 

I would not have "dared" this, but now you suggest it I cannot see anything 
wrong with your approach.


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Some questions about importlib

[Peter Otten]

Quentin Agren wrote:

> Hi,
> 
> My name Quentin, and this is my first post to this list so please redirect
> me if this is not the proper audience.
> 
> I have been studying the 'importlib' standard library package this past
> week, and although I find it very readable I am puzzled by some questions
> that I'd like to share.
> 
> - [Chicken and egg] Does the 'import' statement in Python make use of
> 'importlib'? If so, how are the imports in 'importlib' itself carried out?
> (for example in __init__.py)

Google found

http://sayspy.blogspot.com/2012/02/how-i-bootstrapped-importlib.html

> - Why does 'importlib' use 'nt' or 'psoix' (see
> '_bootstrap_external._setup()') rather than the portable 'os', and
> reimplement some of the latter's functionality, for example '_path_join',
> or '_path_split'. Another example is in 'SourceFileLoader.set_data()'
> where the logic of 'os.mkdirs' is reproduced to create all necessary
> intermediary directories in a file path.
> 
> - Similarly, would not using the 'struct' module simplify the
> packing/unpacking of bytecode. For example by defining BYTECODE_HEADER_FMT
> = '4sII' (Python 3.6)

If the bootstrap process still works as Brett Cannon described it back in 
2012 struct and os cannot be used because they aren't built-in modules.

>>> import sys
>>> "posix" in sys.builtin_module_names
True
>>> "os" in sys.builtin_module_names
False
>>> "struct" in sys.builtin_module_names
False



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] organisation of folder structure of Python's installation directory

[Mats Wichmann]

On 10/21/18 12:52 AM, Malhar Lathkar wrote:
> Hi all.
> 
> I am trying to understand how Python's installation directory is organized.
> I find lib as well as libs folder. what's the difference? I couldn't find
> exact explanation of purpose of various other sub folders like include,
> scripts, share etc. Can you help me understand this? Please share
> documentation link if any.
> 
> thanks.
> 

on a Windows installation, Lib is the python code for the standard
library, while DLLs contains any backing binary objects for the standard
library; meanwhile libs is what is necessary to to link with the python
libraries themselves, as when developing a C-language extension. include
is the header files used for compiling such extensions - so these two
are not part of the Python runtime, but have this special purpose.
(note sometimes when you "python -m pip install" it may try to compile
some code, so this is not always just for you yourself to write extensions).

The rest should be relatively straightforward.

The layout can be quite different on other systems. A /usr/share
directory on Linux is used here for shareable, non-binary pieces like
documentation and license files. The python layout on Linux is normally
packaged along the lines of the Filesystem Hierarchy Standard:

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] What is the best way for a program suite to know where it is installed?

[Oscar Benjamin]

(Resending to the list - sorry for the duplicate, Bob)

On Sun, 21 Oct 2018 at 04:02, boB Stepp  wrote:
>
> I was just re-reading the entire thread at
> https://www.mail-archive.com/tutor@python.org/msg77864.html
> And I have asked similar questions previous to that thread.  I still
> do not have a satisfactory answer for the subject line's question that
> both makes sense to me and seems to be good programming practice.  And
> that works for any OS that the program suite is installed under.
>
> A constantly recurring situation I am having is a programming project
> with a nested directory structure.  Usually I will have "tests" and
> "data" folders nested under the top level project folder.  There may
> be others as well depending on the complexity of the project I am
> attempting.

Is this project a library or an application?

If it is a library then it is important to do things in the way that
users of your library will expect so that they can reasonably install
and use your library or package it up along with other libraries as
part of an application. If it is an application then the requirements
are weaker - it really depends how you package and distribute your
application which is something that is under your control.

> As far as I can tell, I cannot make any assumptions about
> what the current working directory is when a user starts the program.

That depends what kind of an application/library it is and how it is
started. For a library you certainly shouldn't make assumptions about
this. Command line programs that are installed on PATH can be run from
any directory. GUI programs could be run with anything as the current
working directory.

It may not make sense for your program to be installed on PATH though
and you might document that a user should run your program by running
a particular command in a particular directory. There might be other
reasons for doing this such as ensuring that any Python modules are
importable.

Generally speaking though it shouldn't be necessary to depend in any
way on the current working directory just to locate your applications
data files.

> So how can one of my programs *know* what the absolute path is to,
> say, the top level of the program suite?

I assume here that you mean "how can code in a particular Python
module find the base directory that my application/library is
installed in?". The simplest answer is to use __path__/__file__. If
you have many Python files organised into packages and subpackages etc
then you should do this in one place near the top. So if you have

   pgk
   config.py
   data.txt
   subpkg1
submod1.py

then config.py can have the code to check __file__ and work out the
path of the directory containing the pkg folder. The other modules can
all import that from pkg.config.

> If I could always reliably
> determine this by my program's code, every other location could be
> determined from the known nested structure of the program suite.  [But
> I have a dim recollection that either Ben or Cameron thought relying
> on the "known nested structure" is brittle at best.]

One way in which this can go wrong is if you are distributing a
library. It is possible that your library could end up being embedded
in a zip-file so that the files don't exist unzipped on disk.

For the library case it certainly used to be recommended to use pkg_resources to
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access

So for the example above you would use something like:
data_bytes = pkg_resources.resource_string('pkg', 'data.txt')

> So far the best method I've come up with is to make use of "__file__"
> for the initiating program file.  But from past discussions I am not
> certain this is the *best* way.  Is the *best* way going to get me
> into best techniques for installing and distributing programs?  If so,
> this strikes me as a huge mess to dive into!

I think that for a simple application this is fine. You are in control
of how your application is packaged and installed so you can ensure
that this will always work if someone runs your application in the
intended way.

So you can do something like:
project_root = os.path.abspath(os.path.dirname(__file__))
data_file_path = os.path.join(project_root, 'data'.txt')

In the other thread you seemed confused that the effect of abspath (or
similarly realpath) for a given input string depends on the current
directory. That's because the contents of the string __file__ may also
depend on the current working directory. The point of abspath/realpath
is that these effects should cancel so that the resulting string does
not depend on the current working directory.

--
Oscar
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Can tempfile.NamedTemporaryFile(delete=False) be used to create *permanent* uniquely named files?

[Cameron Simpson]

On 21Oct2018 10:55, Peter Otten <__pete...@web.de> wrote:

boB Stepp wrote:

So I am now wondering if using
tempfile.NamedTemporaryFile(delete=False) would solve this problem
nicely?  As I am not very familiar with this library, are there any
unforeseen issues I should be made aware of?  Would this work equally
well on all operating systems?


I think this is cool thinking outside of the box.

I would not have "dared" this, but now you suggest it I cannot see anything
wrong with your approach.


The doco for mktemp (do not use! use mkstemp or the NamedTemporaryFile 
classes instead!) explicitly mentions using delete=False.


Cheers,
Cameron Simpson 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] sftp in python without using pysftp or paramiko

[Steven D'Aprano]

On Sun, Oct 21, 2018 at 12:44:23AM +0100, Alan Gauld via Tutor wrote:
> On 20/10/18 21:07, Asad wrote:
> > hi All ,
> > 
> > I am looking to create a python script which logs in to the server and
> > copies the fie from ftp to local system .:
> 
> The obvious answer is use pysftp.
> But your subject says you don't want to do that.

It could be a good learning exercise.

As you say:

> Reinventing the wheel is a painful
> waste of time that usually only serves to
> increase your respect for the inventor of
> said wheel.

But its not a waste of time if you learn valuable skills from it. 
*Somebody* has the write and maintain the low-level stuff, and if we all 
did nothing but use the high-level libraries, who would maintain them?

I'm reminded of the situation in scientific computing, where there are 
many important libraries written in Fortran which nobody can maintain 
because nobody understands how they work.

Asad: show us the code you've started with. You are far more likely to 
get a useful response to *concrete* questions about a specific problem, 
than a vague "show me what to do" general question.


-- 
Steve
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor