Hermetic environments

[DL Neil]

Is Python going 'the right way' with virtual environments?

Warning: this post may contain Python-heresy.


I don't use any of the Python virtual environments!

I'm not an *ops person either - unless Ops happened to be included 
within a wider department. However, in attempting to follow the "Proper 
shebang for python3" thread,  the following appealed to my mind:

<<<
The truth is that there is no single answer to this problem until you
hermetically seal an environment.  Docker comes close, a freestanding
VM does this most completely.   But short of that, the ability to decide
to use something other than system-provided tools absolutely IS a 
requirement in systems of any scale.

>>>

I do attempt to "hermetically seal" work, building a separate Virtual 
Box VM for each customer (and usually more than one).


To be clear, this is less about Python (I use other languages), and more 
a method of keeping things completely separate, for privacy and 
security. It also allows me to conveniently separate 'dev' from 
'prod'/support.


I don't 'fiddle' but leave all the executables where the OpSys loads 
them (smarter minds than mine!), and load/pip everything as system-wide 
(no thinking necessary!).


With the power of today's machines, I don't really notice* any 
performance degradation. There is no hassle about where the binaries may 
be found. There is no issue if one client is sticking with an earlier 
version but another wants the next iteration of development using some 
v3.7+ advancement. Similarly, there's no chance of tripping-over other 
clients' feet because some are web-based, some GUI, and some cmdLN!


* we recently ran a test on a fairly compute-heavy statistics 
application: the same system on my (relatively old) portable and VBox, 
against a 'delivered version' installed natively on a newer and much 
bigger (RAM and CPU) Windows 10 box. Guess which little, old, 
Fedora-Linux, VBox-slowed machine 'won'!? Whilst the VM imposes an 
overhead 'cost', because it can be 
stripped-down/lean-and-mean/nothing-that's-unnecessary, the VM seems to 
be cost-effective.

(regardless of the above tactics and criteria!)

Standing on principle, I'd like to use a system other than VBox (Oracle) 
but it is neatly cross-platform and multi-platform. When eval-time is 
available, Docker is definitely on the try-out list (as mentioned in the 
quote, above).


Normally, 'delivery' is a matter of duplicating the VM and fitting it 
into the client's DNS. It is similarly easy to pass work to A.N.Other, 
and presents no problem if there are multiple developers. (providing THE 
VM is not (only) on my portable...)



Am I 'getting away with it', perhaps because my work-pattern doesn't 
touch some 'gotcha' or show-stopper?


Why, if so much of 'the rest of the world' is utilising "containers", 
both for mobility and for growth, is the Python eco-system following its 
own path?


Is there something about dev (and ops) using Python venvs which is a 
significant advantage over a language-independent (even better: an 
OpSys-independent) container?


--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: PyPi twine check does not like my README.rst files anymore

[Barry Scott]

I tried the twine check on Fedora and the test pass.

This looks looks like a bug in twine on Windows which I have reported in
https://github.com/pypa/twine/issues/476 


Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Rhodri James]

On 23/07/2019 03:27, DL Neil wrote:

On 23/07/19 11:00 AM, Ethan Furman wrote:

On 07/20/2019 05:02 PM, DL Neil wrote:

Upon closer inspection, I realised it didn't just fail; it failed 
badly! Some silly, little, boy had imported the PythonEnvironment 
class but failed to ALSO import PythonVersionError. So, the reported 
error was not the expected exception!


I don't understand the significance of not importing PythonVersionError:

- PythonEnvironment does not need it to be imported

- when PythonEnvironment raises PythonVersionError you still get 
PythonVersionError


- if your code says `except PythonVersionError` and you haven't 
imported it you'll get a NameError


So, what's the issue?



Have I correctly understood the question?

NameError conveys nothing to the user.
PythonVersionError is more communicative - and speaks volumes to 'us'.

The mainline code is something like:

 p = PythonEnvironment()
 try:
     p.compatibility( ...spec... )    # eg must be Py3 not 2.n
 except PythonVersionError:
     print( more illuminating errmsg )

If I am 'the user' I'd be quite happy without the try...except; but mere 
mortals need/deserve something more. Accordingly, the PythonVersionError 
custom exception/class.


Ah, so you *used* PythonVersionError without importing it.  Sorry, but 
that's on you.


--
Rhodri James *-* Kynesim Ltd
--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Thomas Jollans]

On 23/07/2019 04.27, DL Neil wrote:
> On 23/07/19 11:00 AM, Ethan Furman wrote:
>> On 07/20/2019 05:02 PM, DL Neil wrote:
>>
>>> Upon closer inspection, I realised it didn't just fail; it failed
>>> badly! Some silly, little, boy had imported the PythonEnvironment
>>> class but failed to ALSO import PythonVersionError. So, the reported
>>> error was not the expected exception!
>>
>> I don't understand the significance of not importing PythonVersionError:
>>
>> - PythonEnvironment does not need it to be imported
>>
>> - when PythonEnvironment raises PythonVersionError you still get
>> PythonVersionError
>>
>> - if your code says `except PythonVersionError` and you haven't
>> imported it you'll get a NameError
>>
Actually, if your code says ‘except PythonVersionError’ this will only
raise a NameError if there is an actual exception.

(this was not what I expected - I learned something new today!)

If no exception is being handled, the except statements are never
examined. This makes this harder to spot than you might think!

% cat test.py   
try:
    print('ok')
except FakeError:
    print('ah shit')

% python3 test.py
ok



That makes me think...


def get_exc(exc):
    print("we're checking for exception", exc)
    return exc


try:
    print("raising no. 1")
    raise ValueError
except get_exc(TypeError):
    print("t'was a TypeError")
except get_exc(ValueError):
    print("t'was a ValueError")

try:
    print("raising no. 2")
    raise TypeError
except get_exc(TypeError):
    print("t'was a TypeError")
except get_exc(ValueError):
    print("t'was a ValueError")


>> So, what's the issue?
>
>
> Have I correctly understood the question?
>
> NameError conveys nothing to the user.
> PythonVersionError is more communicative - and speaks volumes to 'us'.
>
> The mainline code is something like:
>
> p = PythonEnvironment()
> try:
>     p.compatibility( ...spec... )    # eg must be Py3 not 2.n
> except PythonVersionError:
>     print( more illuminating errmsg )
>
> If I am 'the user' I'd be quite happy without the try...except; but
> mere mortals need/deserve something more. Accordingly, the
> PythonVersionError custom exception/class.
>
> Yes, we could trap NameError, but that might also catch other
> name-errors (unlikely in this example, but 'just sayin'). Thus the
> greater specificity of the custom class.
>
>
> NB please see alternative 'solution' proposed (for critique) as
> "Nesting Custom Errors in Classes" discussion topic.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[Cameron Simpson]

On 23Jul2019 14:54, DL Neil  wrote:

Do you use nested classes?
Python manages nested classes.

I've NEVER seen such 'in the wild'.
(but perhaps I lead a sheltered life?)


So sheltered :-)

In my experience it's uncommon. I've done it a few times.

"Nested" classes aren't anything very special; they're no more than yet 
another name defined inside a class, like a method, or a "constant".  
Look:


 class Foo:

   DEFAULT_FOO_MAX = 10

   def __init__(...):
 ...

   class SpecialThing:
 ...

They're all just various names in the class space.

I would do it if SpecialThing wasn't meaningful outside of the class 
i.e. didn't really have a standalone use.


Aside from not polluting the top level namespace with names that aren't 
useful on their own (which is kind of irrelevant since people tend to 
"from foo import name", so the _number_ of available names isn't very 
important)...


One potential benefit to to associating the inner class with the outer 
class is the inheritance chain. Look:


 class Foo:

   class ClassyThing:
 ...

   def frob(self, it):
 for i in it:
   x = self.ClassyThing(i)
   x.do_something_classy(...)

 class Bah(Foo):

   class ClassyThing:
 ...

 f = Foo()
 f.frob([1,2,3])

 b = Bah()
 b.frob([1,2,3])

Foo and Bah have a common method "frob", but f.frob will generate and 
use Foo.ClassyThing objects while b.frob will generate Bah.ClassyThing 
objects. Same code.


Personally, I haven't done the above. But I have made subclasses where 
the subclass overrides a constant, like DEFAULT_FOO_MAX earlier. Then 
accessing self.DEFAULT_FOO_MAX finds the appropriate constant.


And on thinking about it, while I haven't explicitly nested classes, I 
have passed in a class to the init method, for example an indexclass.  
I've a package which support several backend hash indices, such as gdbm, 
and I pass in the target index class. So a class as a parameterisable or 
inheritable thing isn't nonsensical.


I have use nested classes when the nested class is strongly associated 
with the enclosing class, like your classname.exceptionname example. I'd 
put it inside the outer class entirely to express that association.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to execute shell command in Python program?

[Madhavan Bomidi]

Thanks for your suggestions.

I have sorted the issue as follows:

import pexpect
child = pexpect.spawn('./opac')
child.sendline('opac')

This works!!!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Ethan Furman]

On 07/22/2019 07:27 PM, DL Neil wrote:


NameError conveys nothing to the user.
PythonVersionError is more communicative - and speaks volumes to 'us'.

The mainline code is something like:

 p = PythonEnvironment()
 try:
     p.compatibility( ...spec... )    # eg must be Py3 not 2.n
 except PythonVersionError:
     print( more illuminating errmsg )


This is why tests should check for the exact error raised, not that any error 
was raised.  (That testing bug bit me a couple times before I learned that 
lesson.)

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Ethan Furman]

On 07/22/2019 08:12 PM, DL Neil wrote:

On 22/07/19 9:40 PM, Thomas Jollans wrote:



Just FYI, in the scientific Python community certain short abbreviations
are the norm. Many modules have a ‘standard’ abbreviation that most
people use, minimizing confusion.

import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt
import pandas as pd
import xarray as xr



The use of meaningful names is a minimum professional standard (IMHO), and in 
my experience, a virtue that pays for itself. I am ever-so grateful that most 
modern text-editors will guess/read-my-mind and save the typing!


It's not just typing, but reading.  Long names, when used over and over, become 
noise.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Creating time stamps

[Peter Pearson]

On Mon, 22 Jul 2019 16:25:32 -0500, Michael F. Stemper wrote:
> On 22/07/2019 15.58, Chris Angelico wrote:
>> On Tue, Jul 23, 2019 at 6:34 AM Michael F. Stemper
>>  wrote:
>>>
[snip]
>>>   from datetime import datetime
>>>   from time import strftime
>>>   timestamp = datetime.now().strftime( "%Y-%m-%d %H:%M" )
[snip]
>> 
>> What's the second import doing though? You never use strftime.
>
> Cleaned my contacts, cleaned my readers. Still see strftime() in the
> third line. Tried to run the code without the second line. It ran
> without complaint.
>
> Apparently, the strftime() in that last line is not the one that I
> explicitly imported, but a method of datetime.now(). Did I get that
> right?

Yes, you got that right.  The statement datetime.now().strftime("...")
says that datetime.now will return an object that has a method
named strftime, and calling that method is the next thing to be done.
Other things that happen to be lying around named strftime just have
no part in the conversation.

-- 
To email me, substitute nowhere->runbox, invalid->com.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[Rob Gaddi]

On 7/22/19 7:54 PM, DL Neil wrote:

Do you use nested classes?

[following-on from the earlier, "Namespaces: memory vs 'pollution'" discussion 
thread, wherein a certain 'someone' remembered to from ... import ... as ... an 
'action' class but forgot to also import the related custom error class! The 
original quest was for a wild-card import device. This discussion may obviate 
continuing the quest and/or needing to remember...]



Python manages nested classes.

I've NEVER seen such 'in the wild'.
(but perhaps I lead a sheltered life?)


What are proposed as use-cases for such a technique?
- other languages may not offer decent "inheritance", and this is an alternative 
method/hack

- the class is a 'class factory', generating/returning an object
? any others


Why not use it as an "encapsulation" device?
(please be gentle - reminder: I am too old to have been an OO-native!)


* stub definitions

 >>> class PythonEnvironment():
... class PythonVersionError( EnvironmentError ):
... pass
... def iscompatible( self ):
     ''' Ensure that the Python in-use will support
     all of the facilities employed by the application.
     '''
... # stub to simulate failure
... raise self.PythonVersionError
...

(code would require an import or from ... import ...)
 >>> pe = PythonEnvironment()


* its basic application becomes:-

 >>> pe.iscompatible()
Traceback (most recent call last):
   File "", line 1, in 
   File "", line 6, in compatibility
__main__.PythonVersionError


* somewhat more realistic use:-

 >>> try:
... pe.iscompatible()
... except PythonEnvironment.PythonVersionError:
... print( "Trapped! -> informative errmsg" )
...
Trapped! -> informative errmsg


With this construct, one only has to import the 'outer' class, rather than both 
the class AND its ancillary error class!



Why haven't I seen it before? Can you see anything 'wrong' with this picture?


I've used them sometimes for basic encapsulation principles without really 
gaining anything.  I use inheritance of nested classes in my 
https://pypi.org/project/indexedproperty/ project, where subclasses of 
IndexedProperty have a _Trampoline inner class subclassed from Trampoline that 
can be redefined as needed, but that's a bit of an obscure use case.


--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[DL Neil]

On 23/07/19 11:55 PM, Cameron Simpson wrote:

On 23Jul2019 14:54, DL Neil  wrote:

Do you use nested classes?
Python manages nested classes.

I've NEVER seen such 'in the wild'.
(but perhaps I lead a sheltered life?)


So sheltered :-)


Well, over here there aren't the venomous creatures you live amongst on 
'west island'...




In my experience it's uncommon. I've done it a few times.

"Nested" classes aren't anything very special; they're no more than yet 
another name defined inside a class, like a method, or a "constant". Look:



Thanks for the explanation. I've absorbed the concept, but back in the 
?good, old days it was common practice to alter code programmatically, 
ie whilst the pgm was running. The perils of not quite knowing what the 
code 'is' and thus what it should be doing for us, constitute a 
debugging nightmare. (but hey, even flicking toggle-switches on the 
front of an IBM System/360 to load binary data into a specified memory 
address, seemed like 'great power' - at the time *cue SpiderMan quote)


Accordingly, the idea that there would be a ClassyThing() but it might 
be quite different from another ClassyThing() just makes me shudder!

(sorry, I guess it's that sheltered upbringing ... again!)

As you say...

Personally, I haven't done the above. But I have made subclasses where 
the subclass overrides a constant, like DEFAULT_FOO_MAX earlier. Then 
accessing self.DEFAULT_FOO_MAX finds the appropriate constant.


And on thinking about it, while I haven't explicitly nested classes, I 
have passed in a class to the init method, for example an indexclass. 
I've a package which support several backend hash indices, such as gdbm, 
and I pass in the target index class. So a class as a parameterisable or 
inheritable thing isn't nonsensical.


I have use nested classes when the nested class is strongly associated 
with the enclosing class, like your classname.exceptionname example. I'd 
put it inside the outer class entirely to express that association.


...and, because the (?old) boy can't be relied upon to remember to 
import both, nesting saves-the-day!


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Embedding Python in C

[Stefan Behnel]

Jesse Ibarra schrieb am 22.07.19 um 18:12:
> On Saturday, July 20, 2019 at 1:11:51 PM UTC-6, Stefan Behnel wrote:
>> Jesse Ibarra schrieb am 20.07.19 um 04:12:
>>> Sorry, I am not understanding. Smalltlak VW 8.3 does not support Python.
>>> I can only call Pyhton code through C/Python API.
>>
>> Ok, but that doesn't mean you need to write code that uses the C-API of
>> Python. All you need to do is:
>>
>> 1) Start up a CPython runtime from Smalltalk (see the embedding example I
>> posted) and make it import an extension module that you write (e.g. using
>> the "inittab" mechanism [1]).
>>
>> 2) Use Cython to implement this extension module to provide an interface
>> between your Smalltalk code and your Python code. Use the Smalltalk C-API
>> from your Cython code to call into Smalltalk and exchange data with it.
>>
>> Now you can execute Python code inside of Python and make it call back and
>> forth into your Smalltalk code, through the interface module. And there is
>> no need to use the Python C-API for anything beyond step 1), which is about
>> 5 lines of Python C-API code if you write it yourself. Everything else can
>> be implemented in Cython and Python.
>>
>> Stefan
>>
>>
>> [1]
>> https://docs.python.org/3/extending/embedding.html?highlight=PyImport_appendinittab#extending-embedded-python
> 
> This cleared so much @Stefan, thank you. I just need some clarification if 
> you don't mind.
>  
> In (1), when you say  "import an extension module that you write",  do you 
> mean the Python library that was created "import emb"? Is that gonna be 
> written in Cython or standalone .C file?

Yes. In Cython.


> in (2), what do to mean when you said "Use the Smalltalk C-API from your 
> Cython code to call into Smalltalk and exchange data with it."? 

Not sure what part exactly you are asking about, but you somehow have to
talk to the Smalltalk runtime from your Cython/Python code if you want to
interact with it. I assume that this will be done through the C API that
Smalltalk provides.

Just in case, did you check if there is already a bridge for your purpose?
A quick web search let me find this, not sure if it helps.

https://github.com/ObjectProfile/PythonBridge

Stefan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[Cameron Simpson]

On 24Jul2019 07:21, DL Neil  wrote:
Accordingly, the idea that there would be a ClassyThing() but it might 
be quite different from another ClassyThing() just makes me shudder!

(sorry, I guess it's that sheltered upbringing ... again!)


It needn't be very different on the outside. The whole point of object 
oriented programme is to present things with the same external API but 
different internal implementations, or managing a different thing with 
common characteristics.


I have use nested classes when the nested class is strongly associated 
with the enclosing class, like your classname.exceptionname example. 
I'd put it inside the outer class entirely to express that 
association.


...and, because the (?old) boy can't be relied upon to remember to 
import both, nesting saves-the-day!


Get some linting tools. They're great for catching this kind of error.

The tricky bit with dynamic language like Python is that some naming 
errors (missed imports) aren't apparent until the code passes through 
that path. Linters can cover a lot of this with static analysis.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[DL Neil]

On 24/07/19 10:07 AM, Cameron Simpson wrote:

On 24Jul2019 07:21, DL Neil  wrote:

...

Get some linting tools. They're great for catching this kind of error.


SublimeText has SublimeLinter and PycodeStyle installed, but I'm still 
familiarising myself with ST. Nothing is reported.


Any advice/correction/improvement would be most welcome...


The tricky bit with dynamic language like Python is that some naming 
errors (missed imports) aren't apparent until the code passes through 
that path. Linters can cover a lot of this with static analysis.


Exactly!
(PyTest and coverage tools were no help - they import correctly (and 
test adequately), but I didn't open those (separate) modules to notice 
the multiple import-s)


Hope springs eternal ... there must be a better way???

(well yes, not wishing to 'flog a dead horse' (nor any horse for that 
matter) I'm pleased at the (lack of negative) response to my allied 
question about nested classes, which would solve the original problem)


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Nesting Custom Errors in Classes

[Cameron Simpson]

On 24Jul2019 11:47, DL Neil  wrote:

On 24/07/19 10:07 AM, Cameron Simpson wrote:

On 24Jul2019 07:21, DL Neil  wrote:

...

Get some linting tools. They're great for catching this kind of error.


SublimeText has SublimeLinter and PycodeStyle installed, but I'm still 
familiarising myself with ST. Nothing is reported.


Any advice/correction/improvement would be most welcome...


Hmm. I'd expect a linter to catch this:

 # forgot FooException
 from foo import FooClass
 def f():
   raise FooException

Can you test that with a small script at your end (with real imports and 
names of course)?


I lint from the command line with this script:

 https://bitbucket.org/cameron_simpson/css/src/tip/bin-cs/lint

which runs a few Python linters for Python.

WRT SublimeText, PycodeStyle likely only checks style (I've now shifted 
to an automatic formatter for this) and SublimeLinter's PythonLinter 
page is... silent; it looks like a base class for other actual linters.  
You might need to do some more digging. Maybe you need some addition 
lint stuff for SublimeText.


The tricky bit with dynamic language like Python is that some naming 
errors (missed imports) aren't apparent until the code passes through 
that path. Linters can cover a lot of this with static analysis.


Exactly!
(PyTest and coverage tools were no help - they import correctly (and 
test adequately), but I didn't open those (separate) modules to notice 
the multiple import-s)


Hope springs eternal ... there must be a better way???


A good linter is usually decent at complaining about code using names 
that aren't defined (not imports, not otherwise defined such as a 
function or a variable assignment).


(well yes, not wishing to 'flog a dead horse' (nor any horse for that 
matter) I'm pleased at the (lack of negative) response to my allied 
question about nested classes, which would solve the original problem)


There's nothing "wrong" with it. It is uncommon. But like a lot of 
things, the value (or cost/pitfalls) come with how things are to be 
used.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list