Re: WedWonder: Scripts and Modules

[Barry Scott]

> On 11 Sep 2019, at 21:24, DL Neil via Python-list  
> wrote:
> 
> In this day-and-age do you have a script in live/production-use, which is 
> also a module? What is the justification/use case?
> 
> (discounting distutils and similar installation tools, or unit testing 
> methodology)
> 
> 
> There are over 500 questions on StackOverflow which refer to Python's
> 
>   if __name__ == __main__:
> 
> construct. Even more if you include the idea of a main() multiple entry-point.
> 
> This construct enables code to distinguish between being "run" as a "script", 
> and being imported as a "module". (which is not in question)
> 

In my mind this is a question about what side-effects of importing a module are
desireable and which are not.

A trivia script does not need the __name__ == '__main__' as its all about its 
side effects.

As scripts become more complex having it run on import might make debugging 
harder
and prevents reuse.

For example I will import a script at the REPL and examine it and call function 
in it to
help me understand and fix problems.  Having a __name__ == '__main__' is 
important
to allow this.

I often have modules that are part of a larger program that have their own 
main() functions
to unittest or give access to parsers etc.

In large projects with many modules import with side-effect can make for a 
maintenance
burden.

Barry


> -- 
> Regards,
> =dn
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Email messages from grouped email using IMAPClient in Python.

[Barry Scott]

> On 12 Sep 2019, at 06:06, Srinivas Pullabhotla  wrote:
> 
> Hello,
> 
> I am trying to fetch email messages from a gmail inbox. So, there will be 
> 1000s of messages sent to Inbox and since they are 1000s, the emails are 
> grouped 100 per each email item.
> 
> When I tried this method, the program only fetches some but not all and 
> especially it happens with grouped email messages. How can I get all the 
> messages from that grouped email.
> 
> import email, time, sys
>from imapclient import IMAPClient
> 
>with IMAPClient(HOST) as server:
>server.login(USERNAME, PASSWORD)
>server.select_folder('INBOX', readonly=True)
> 
>messages = server.search(['ALL', 'UNSEEN'])
>for uid, message_data in server.fetch(messages, 'RFC822').items():
>email_message = email.message_from_bytes(message_data[b'RFC822'])
>print(email_message.get_payload(None, True))
> 
> 
> The program should fetch all the email messages from the grouped email and 
> output to the file (in my case I am grabbing the href links). How best I can 
> achieve this ? Appreciate thoughts and suggestions.

If you have a very large number of messages then you will need to add:

# default of 1 is too small for SEARCH results
imaplib._MAXLINE = 20

If you want to use the MOVE command you will need this:

# MOVE is not available by default
imaplib.Commands['MOVE'] = ('SELECTED',)

As DL said you need to print out what you get back from the search() and check 
that it is what you expect.

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[Chris Angelico]

On Thu, Sep 12, 2019 at 6:57 PM Barry Scott  wrote:
> As scripts become more complex having it run on import might make debugging 
> harder
> and prevents reuse.
>
> For example I will import a script at the REPL and examine it and call 
> function in it to
> help me understand and fix problems.  Having a __name__ == '__main__' is 
> important
> to allow this.

If the script is fairly simple, and hasn't been built to operate as a
module, it's often effective enough to just run "python3 -i
scriptname.py" and have it do its stuff and immediately drop to REPL.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Error

[Courtney Mark]

Hi

I am a brand new user.

I cannot type onto a shell window or a code window.

Kind Regards
Courtney

Sent from Mail for Windows 10

-- 
https://mail.python.org/mailman/listinfo/python-list

numpy results in segmentation fault

[Pradeep Patra]

Hi ,

I was trying to solve the hackerrank and was using python 3.7.x.
https://www.hackerrank.com/challenges/np-concatenate/problem

While running the code sometimes I get success result and sometimes it
fails with "Segmentation Fault" at Hacker rank UI. I dont have any clue why
the code is crashing ? Does anyone have any idea?

Regards
Pradeep

import numpy

n,m,p = map(int,input().split())
tgt_arr1 = []
for i in range(n):
row = list(map(int,input().split()))
tgt_arr1.append(row)
tgt_arr2 = []
for j in range(m):
row = list(map(int,input().split()))
tgt_arr2.append(row)

num_arr1 = numpy.array(tgt_arr1,int)
num_arr2 = numpy.array(tgt_arr2,int)

print(numpy.concatenate((num_arr1,num_arr2),axis=0))
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: numpy results in segmentation fault

[Thomas Jollans]

On 12/09/2019 15.53, Pradeep Patra wrote:
> Hi ,
>
> I was trying to solve the hackerrank and was using python 3.7.x.
> https://www.hackerrank.com/challenges/np-concatenate/problem
>
> While running the code sometimes I get success result and sometimes it
> fails with "Segmentation Fault" at Hacker rank UI. I dont have any clue why
> the code is crashing ? Does anyone have any idea?


Are you sure it's your code that's crashing, and not something beyond
your control? (Such as the software that is starting Python for you)
Does it depend on the input? Can you reproduce the issue in a controlled
environment (i.e. on your own PC)?


>
> Regards
> Pradeep
>
> import numpy
>
> n,m,p = map(int,input().split())
> tgt_arr1 = []
> for i in range(n):
> row = list(map(int,input().split()))
> tgt_arr1.append(row)
> tgt_arr2 = []
> for j in range(m):
> row = list(map(int,input().split()))
> tgt_arr2.append(row)
>
> num_arr1 = numpy.array(tgt_arr1,int)
> num_arr2 = numpy.array(tgt_arr2,int)
>
> print(numpy.concatenate((num_arr1,num_arr2),axis=0))


-- 
https://mail.python.org/mailman/listinfo/python-list

cx_freeze bdist_msi: shortcut in startmenu sub-folder?

[Grant Edwards]

I've got a couple tk apps I distribute for windows using cx_feeze.
Currently I use Inno Setup to create a setup.exe, but would like to
switch to using cx_freeze's msi feature.

But, I'm stumped trying to figure out how to create an entry in a
folder underneath the Windows startmenu.

The cx_freeze initial_target_dir option allows you to specify a folder
path like this:

"initial_target_dir": 
r"[ProgramFilesFolder]\my-company-name\my-app-name"

But, cx_freeze's shortcutDir option doesn't seem to accept a path like
that, so I'm only able to create an entry in the top-level start menu.

Can anybody point me to an example or clue on how to create a shortcut
in a folder underneath the start menu with cx_freeze bdist_msi?

-- 
Grant Edwards   grant.b.edwardsYow! We are now enjoying
  at   total mutual interaction in
  gmail.coman imaginary hot tub ...

-- 
https://mail.python.org/mailman/listinfo/python-list

Scientific Software Developers | Job positions at CMCC Foundation - Euro-Mediterranean Center on Climate Change, Italy

[CMCC Info]

/Please, feel free to circulate //to anyone you think may be interested.///
--
Open positions at CMCC Foundation:

*Software Developers for the Digital Ocean (Research Associate) 
* 
[Job Opening Code: 9195]
The location is the CMCC – Ocean Predictions and Applications (OPA) 
office in Lecce, Italy.

_Deadline: 15/09/2019 _

*Scientific Software Developer 
***[Job 
Opening Code: 9209]
The location is CMCC Headquarters in Lecce, Italy. Remote working as 
well as part-time could be suitable options.

_Deadline: 20/09/2019 _

*Software Developer for the Digital Ocean (Research Associate) 
* 
[Job Opening Code: 9213]
The location is CMCC – Ocean Predictions and Applications (OPA) office 
in Bologna (Italy)

_Deadline: 20/09/2019 _

For further information on CMCC Job Opportunities, please visit our 
website https://www.cmcc.it/work-with-us .


--

Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici
Web: www.cmcc.it  - Contact us: i...@cmcc.it 



--
https://mail.python.org/mailman/listinfo/python-list

duck typing / library functions

[duncan smith]

Hello,
  The problem I have relates to writing algorithmic code that can
handle types with a given API, but where some of the required
functionality is implemented as library functions rather than methods.
Specifically I have code that uses numpy arrays, but I want to adapt it
to use sparse arrays that have a (sufficiently) similar API.

For the most part the existing code will work without alteration, but
wherever I have a numpy function (rather than array method) I have an
issue. In some cases I can find a corresponding method, but not for e.g.
numpy.isnan. Roughly the structure of the existing code is:



import numpy


class Algorithm(object):
def __init__(self, arrays):
self.arrays = arrays

def run(self):
shape = a shape calculated from info in self.arrays
arr = numpy.ones(shape)
# other stuff using array methods and numpy functions



I could replace "numpy" above with "sparse" and it would work fine (as
long as the arrays passed to __init__ were of the appropriate type). I
could move the import inside the class and make it conditional on the
types in self.arrays. I could potentially replace the function calls
with something based only on method calls (although that turns out to be
awkward for some functions). But I'm thinking about something more
introspective, maybe identifying the relevant module / package from the
array type and importing the relevant module on the fly? Any ideas? TIA.

Duncan

p.s. Ideally the solution should work with Python 3 and recent versions
of Python 2.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[DL Neil via Python-list]

On 12/09/19 10:59 AM, Cameron Simpson wrote:

On 12Sep2019 08:24, DL Neil  wrote:
In this day-and-age do you have a script in live/production-use, which 
is also a module? What is the justification/use case?


Many. Many many.

1: Many of my modules run their unit tests if invoked as the main 
programme.


I used to do this, but now prefer to keep tests in separate modules - 
and separate directories.



2: Several modules are their own utility programme. I've got a heap of 
these - anything that provides useful access to something can often be 
usefully used from the command line.


This is an very interesting idea - you're way ahead of me on this!

Would it be fair to describe these as more Python for systems 
programming/system administration, than an application for (simple) users?

(what other kind is there???)



Consider: if you write a package, would it have a __main__.py?

Well, if the answer is ever "yes" then the same applies to ordinary 
modules, simple enough to not be worth splitting onto a package.


May not properly appreciate this point...



So, yes, for me this is really really common.

Even for my current client project, which is largely a package, several 
of the individual modules within the package have their own main 
programmes for testing and for various utility tasks dealing solely with 
that particular subsystem. There's an overarching shell script to set up 
the environment and then do various things from the command line, and it 
directly invokes particular modules for some operations that act only on 
one subsystem.


Interesting!

Gives me pause for thought: perhaps I lean too heavily on putting 
'stuff' in the test routines (and view the application from that 
'direction' too often).


Multiple entry-point systems seem relatively unusual these days - 
perhaps a natural flow-on effect of the rise of gui- and menu-based systems?


With one client, over the years, we've developed a number of (basically) 
statistical analyses. Each analysis was born from a separate project. 
Each lives in its own directory (tree). There are some common modules 
held in a separate 'utility' library/package/directory. Whilst it has 
been suggested, the idea of an "overarching" menu-type top-level 
controller/distributor has never risen up the ranks of the "backlog" 
(sensible criticism: the money would be better spent on other 
priorities) - they (or at least the older ones) are quite happy to 
'drop' to the command line and type the required command+args. I think 
that's the closest I come to what you have described. (out of interest) 
Would you characterise it as a common project structure?


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[DL Neil via Python-list]

On 12/09/19 8:22 PM, Barry Scott wrote:




On 11 Sep 2019, at 21:24, DL Neil via Python-list  
wrote:

In this day-and-age do you have a script in live/production-use, which is also 
a module? What is the justification/use case?

(discounting distutils and similar installation tools, or unit testing 
methodology)


There are over 500 questions on StackOverflow which refer to Python's

if __name__ == __main__:

construct. Even more if you include the idea of a main() multiple entry-point.

This construct enables code to distinguish between being "run" as a "script", and being 
imported as a "module". (which is not in question)



In my mind this is a question about what side-effects of importing a module are
desireable and which are not.


Precise description!



A trivia script does not need the __name__ == '__main__' as its all about its 
side effects.

As scripts become more complex having it run on import might make debugging 
harder
and prevents reuse.


Is this an informal distinction: that modules are written for re-use but 
main-lines to be unique/single-purpose?




For example I will import a script at the REPL and examine it and call function 
in it to
help me understand and fix problems.  Having a __name__ == '__main__' is 
important
to allow this.


Why?

If we import sys (from the PSL, purely as an example), we don't 
expect/need any execution phase, can immediately follow (in the REPL) 
with help(sys) and similar, and can debug/explore from there:



import sys
help(sys)



sys.path
['', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', 
'/usr/lib64/python3.7/lib-dynload', 
'/usr/lib64/python3.7/site-packages', '/usr/lib/python3.7/site-packages']




I often have modules that are part of a larger program that have their own 
main() functions
to unittest or give access to parsers etc.


and the main() is the sole content of the if __name__ etc structure?



In large projects with many modules import with side-effect can make for a 
maintenance
burden.



You seem to have unmasked an assumption under which I operate. (well 
done - you know what 'they' say about assumptions!)


As you say, (non-designed) side-effects are undesirable.

My 'rule' is that modules only contain definitions, eg classes and 
functions. Thus, *nothing* executes upon import.


During a code review, some eagle-eye, noticed (and questioned) I had 
re-factored some 'constants' which control a "switch" structure from out 
of the module-space, and into the class where they 'belonged', without 
having any other 'good reason'.



I have no recollection of the history or rationale for this 
policy/practice, nor can I remember whether it 'belongs to Python' or 
some other language from which I've imported it (hah, punny joke!) 
Perhaps it comes from my preference to from...import... rather than 
import...as... which largely eschews the module namespace?

(or perhaps I'm just a closet control-freak?)

That said, I'm sure there must be modules of my code which break this 
'rule', somewhere (certainly in the 'quick and dirty' department).


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[Cameron Simpson]

On 13Sep2019 08:40, DL Neil  wrote:

On 12/09/19 10:59 AM, Cameron Simpson wrote:

On 12Sep2019 08:24, DL Neil  wrote:
In this day-and-age do you have a script in live/production-use, 
which is also a module? What is the justification/use case?


Many. Many many.

1: Many of my modules run their unit tests if invoked as the main 
programme.


I used to do this, but now prefer to keep tests in separate modules - 
and separate directories.


My tests are also in a separate module, but I can invoke them from the 
primary module. Just convenience.


2: Several modules are their own utility programme. I've got a heap of 
these - anything that provides useful access to something can often be 
usefully used from the command line.


This is an very interesting idea - you're way ahead of me on this!

Would it be fair to describe these as more Python for systems 
programming/system administration, than an application for (simple) 
users?


Perhaps. I'm not sure this is a meaningful distinction, except in so far 
as simple users generally expecting a GUI, or programmers expecting an 
API instead of a command line.


Having a command line, particularly one doing the common pattern of 
"command op args..." with various "op"s for common simple tasks makes it 
easier to use the module in a script, such as a non-Python script such 
as the shell.


That said, half these "op"s tend to be admin tasks or simple 
integration-like tests. So "make a new app user record in the database" 
=> "command new-user ...". But also "test this application function" ==> 
"command do-thing ..." where "do-thing" is something I'm debugging or 
testing which is normally painfully buried in the module or at the end 
of some tedious user interaction. And of course "command selftest" to 
run the unit tests :-)


Gives me pause for thought: perhaps I lean too heavily on putting 
'stuff' in the test routines (and view the application from that 
'direction' too often).


My weakness lies in the other direction - not enough test driven 
development. IMO, my test writing ability is weak. So I've got this 
chunk of code and now (==later) I want to test some part of it.


Things I am finding useful recently include:

- doctests (which have the advantage that I can also use them as example 
 code in the docstrings) which are nice for laying out common use and 
 also specific corner cases


- the icontract Python module, which lets me put @require and @ensure 
 decorators on functions to express preconditions and postconditions;
 really great for catching misuse, especially if you also put some 
 insinstance type checks in there for getting parameters misordered


Both of these lack the cumbersomeness of a unit test suite. OTOH, they 
do not lend themselves to complex tests.


Multiple entry-point systems seem relatively unusual these days - 
perhaps a natural flow-on effect of the rise of gui- and menu-based 
systems?


Unsure. Quite possibly.

With one client, over the years, we've developed a number of 
(basically) statistical analyses. Each analysis was born from a 
separate project. Each lives in its own directory (tree). There are 
some common modules held in a separate 'utility' 
library/package/directory. Whilst it has been suggested, the idea of 
an "overarching" menu-type top-level controller/distributor has never 
risen up the ranks of the "backlog" (sensible criticism: the money 
would be better spent on other priorities) - they (or at least the 
older ones) are quite happy to 'drop' to the command line and type the 
required command+args. I think that's the closest I come to what you 
have described. (out of interest) Would you characterise it as a common 
project structure?


It certainly seems reasonable. And as you suggest, there's you've got a 
bunch of main programmes associated with distinct modules/packages.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: WedWonder: Scripts and Modules

[Richard Damon]

On 9/12/19 5:11 PM, DL Neil via Python-list wrote:
> On 12/09/19 8:22 PM, Barry Scott wrote:
>
>> In large projects with many modules import with side-effect can make
>> for a maintenance
>> burden.
>
>
> You seem to have unmasked an assumption under which I operate. (well
> done - you know what 'they' say about assumptions!)
>
> As you say, (non-designed) side-effects are undesirable.
>
> My 'rule' is that modules only contain definitions, eg classes and
> functions. Thus, *nothing* executes upon import.
>
> During a code review, some eagle-eye, noticed (and questioned) I had
> re-factored some 'constants' which control a "switch" structure from
> out of the module-space, and into the class where they 'belonged',
> without having any other 'good reason'.
>
>
> I have no recollection of the history or rationale for this
> policy/practice, nor can I remember whether it 'belongs to Python' or
> some other language from which I've imported it (hah, punny joke!)
> Perhaps it comes from my preference to from...import... rather than
> import...as... which largely eschews the module namespace?
> (or perhaps I'm just a closet control-freak?)
>
> That said, I'm sure there must be modules of my code which break this
> 'rule', somewhere (certainly in the 'quick and dirty' department).
>
Well technically, def and class are statements that are executed when
the module is imported, you need to execute them to add the definitions
into the appropriate namespace.


Perhaps the 'rule' needs to be somewhat restated to a form where the
only thing that should execute when importing a module are the
statements needed to create the definitions for the module.


-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Friday Finking: 'main-lines' are best kept short

[DL Neil via Python-list]

(this follows some feedback from the recent thread: "WedWonder: Scripts 
and Modules" and commences a somewhat-related topic/invitation to 
debate/correct/educate)



Is it a good idea to keep a system's main-line* code as short as 
possible, essentially consigning all of 'the action' to application and 
external packages and modules?


* my choice of term: "main-line", may be taken to mean:
- the contents of main(),
- the 'then clause' of an if __name__ == __main__: construct,
- a __main__.py script.


In a previous thread I related some ?good, old days stories. When we 
tried to break monolithic programs down into modular units, a 'rule of 
thumb' was "one page-length" per module (back in the mainframe days our 
code was 'displayed' on lineflo(w) (continuous stationery) which was 66 
- call it 60, lines per page - and back-then we could force a page-break 
where it suited us!). Then when we moved to time-share screens (80 
characters by 24 lines), we thought that a good module-length should 
conform to screen-size. These days I have a large screen mounted in 
'portrait mode', so on that basis I'm probably back to 50~60 lines (yes, 
these old eyes prefer a larger font - cue yet more cheeky, age-ist 
comments coming from my colleagues...)


Likely I have also picked-up and taken-to-heart the *nix mantra of code 
doing 'one job, and doing it well' (and hence the extensive powers of 
redirects, piping, etc - in Python we 'chain' code-units together with 
"import"). Accordingly, I tend to err on the side of short units of 
code, and thus more methods/functions than others might write.


In "Mastering Object-oriented Python" the author discusses "Designing a 
main script and the __main__ module" (Ch17):

<<<
A top-level main script will execute our application. In some cases, we 
may have multiple main scripts because our application does several 
things. We have three general approaches to writing the top-level main 
script:
• For very small applications, we can run the application with python3.3 
some_script.py . This is the style that we've shown you in most examples.
• For some larger applications, we'll have one or more files that we 
mark as executable with the OS chmod +x command. We can put these 
executable files into Python's scripts directory with our setup.py 
installation. We run these applications with some_script.py at the 
command line.

• For complex applications, we might add a __main__.py module in the
application's package. To provide a tidy interface, the standard library
offers the runpy module and the -m command-line option that will use 
this specially named module. We can run this with python3.3 -m some_app.

[explanation of "shebang" line - the second approach, above]

Creating a __main__ module
To work with the runpy interface, we have a simple implementation. We 
add a small __main__.py module to our application's top-level package. 
We have emphasized the design of this top-level executable script file.
We should always permit refactoring an application to build a larger, 
more sophisticated composite application. If there's functionality 
buried in __main__.py , we need to pull this into a module with a clear, 
importable name so that it can be used by other applications.

A __main__.py module should be something small like the following code:

import simulation
with simulation.Logging_Config():
with simulation.Application_Config() as config:
main= simulation.Simulate_Command()
main.config= config
main.run()

We've done the minimum to create the working contexts for our 
application. All of the real processing is imported from the package. 
Also, we've assumed that this __main__.py module will never be imported.
This is about all that should be in a __main__ module. Our goal is to 
maximize the reuse potential of our application.

[example]

We shouldn't need to create composite Python applications via the 
command-line API. In order to create a sensible composition of the 
existing applications, we might be forced to refactor stats/__main__.py 
to remove any definitions from this module and push them up into the 
package as a whole.

>>>

Doesn't the author thus suggest that the script (main-line of the 
program) should be seen as non-importable?


Doesn't he also suggest that the script not contain anything that might 
be re-usable?


Accordingly, the script calls packages/modules which are both importable 
and re-usable.


None of which discounts the possibility of having other 'main-lines' to 
execute sub-components of the (total) application, should that be 
appropriate.


An issue with 'main-line' scripts is that they can become difficult to 
test - or to build, using TDD and pytest (speaking personally). Pytest 
is great for unit tests, and can be used for integration testing, but 
the 'higher up' the testing pyramid we go, the less effectual it becomes 
(please don't sho