Displaying CPU instruction sets used for TensorFlow build?
Hi, Does anyone know how I can display the CPU instruction sets which were used when TensorFlow was compiled? I initially compiled TF on a machine with a CPU which supports AVX512_VNNI. I subsequently recompiled on a second machine without AVX512_VNNI, but when I run a test program on the second machine, I get the error: The TensorFlow library was compiled to use AVX512_VNNI instructions, but these aren't available on your machine. I would like to check the instruction sets explicitly, so I can tell whether I am using the version of TF I think I am, or whether the test program has some sort of problem. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Installing package as root to a system directory
Hi, I use poetry to develop system software packages as a normal user. To install the packages I use, again as a normal user export PYTHONUSERBASE=/some/path pip3 install --user somepackage.whl and add /some/path to /usr/lib64/python3.6/site-packages/zedat.pth This works well enough, but seems to me to be a little clunky, mainly because the files don't then belong to root. The most correct way, in my case, would probably be to create an RPM out of the Python package, but that seems like it would be too much overhead. What other approaches to people use? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
NameError: name '__version__' is not defined
Hi, I have two applications. One uses the system version of Python, which is 3.6.8, whereas the other uses Python 3.10.8 installed in a non-system path. For both applications I am using poetry with a pyproject.toml file which contains the version information and __init__.py at the root which contains try: import importlib.metadata as importlib_metadata except ModuleNotFoundError: import importlib_metadata __version__ = importlib_metadata.version(__name__) For the application with the system Python this mechanism works, but for the non-system Python I get the error: NameError: name '__version__' is not defined For the 3.6 application I have PYTHONPATH=/nfs/local/lib/python3.6/site-packages PYTHONUSERBASE=/nfs/local PYTHON_VERSION=3.6 PYTHON_VIRTUALENV= and for the 3.10 application I have PYTHONPATH=/nfs/easybuild/software/Python/3.10.8-GCCcore-12.2.0/easybuild/python:/nfs/local/lib/python3.10/site-packages PYTHONUSERBASE=/nfs/local PYTHON_VERSION=3.10 PYTHON_VIRTUALENV= The applications are installed in /nfs/local/lib/python3.6/site-packages and /nfs/local/lib/python3.10/site-packages, respectively. Can anyone see where this is going wrong? I thought it should be enough that the packages with the metadata is available via PYTHONPATH, but this seems not to be sufficient. So I must be overseeing something. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: NameError: name '__version__' is not defined
"Loris Bennett" writes:
> Hi,
>
> I have two applications. One uses the system version of Python, which
> is 3.6.8, whereas the other uses Python 3.10.8 installed in a non-system
> path. For both applications I am using poetry with a pyproject.toml
> file which contains the version information and __init__.py at the root
> which contains
>
> try:
> import importlib.metadata as importlib_metadata
> except ModuleNotFoundError:
> import importlib_metadata
>
> __version__ = importlib_metadata.version(__name__)
>
> For the application with the system Python this mechanism works, but for
> the non-system Python I get the error:
>
> NameError: name '__version__' is not defined
>
> For the 3.6 application I have
>
> PYTHONPATH=/nfs/local/lib/python3.6/site-packages
> PYTHONUSERBASE=/nfs/local
> PYTHON_VERSION=3.6
> PYTHON_VIRTUALENV=
>
> and for the 3.10 application I have
>
>
> PYTHONPATH=/nfs/easybuild/software/Python/3.10.8-GCCcore-12.2.0/easybuild/python:/nfs/local/lib/python3.10/site-packages
> PYTHONUSERBASE=/nfs/local
> PYTHON_VERSION=3.10
> PYTHON_VIRTUALENV=
>
> The applications are installed in /nfs/local/lib/python3.6/site-packages
> and /nfs/local/lib/python3.10/site-packages, respectively.
>
> Can anyone see where this is going wrong? I thought it should be
> enough that the packages with the metadata is available via PYTHONPATH,
> but this seems not to be sufficient. So I must be overseeing something.
If in the 3.10 application I add
print(f"__init__ Version: {__version__}")
to __init__.py the correct version is printed. So the problem is that
the variable is not available at the point I am trying access it. The
relevant code (a far as I can tell) in main.py looks like this:
import typer
app = typer.Typer()
@app.callback()
def version_callback(value: bool):
if value:
typer.echo(f"Version: {__version__}")
raise typer.Exit()
@app.callback()
def common(
ctx: typer.Context,
version: bool = typer.Option(None, "--version",
help="Show version",
callback=version_callback),
):
pass
if __name__ == "__main__":
app()
This is the first time I have used typer, so it is more than likely that
I have made some mistakes.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: NameError: name '__version__' is not defined
"Loris Bennett" writes:
> "Loris Bennett" writes:
>
>> Hi,
>>
>> I have two applications. One uses the system version of Python, which
>> is 3.6.8, whereas the other uses Python 3.10.8 installed in a non-system
>> path. For both applications I am using poetry with a pyproject.toml
>> file which contains the version information and __init__.py at the root
>> which contains
>>
>> try:
>> import importlib.metadata as importlib_metadata
>> except ModuleNotFoundError:
>> import importlib_metadata
>>
>> __version__ = importlib_metadata.version(__name__)
>>
>> For the application with the system Python this mechanism works, but for
>> the non-system Python I get the error:
>>
>> NameError: name '__version__' is not defined
>>
>> For the 3.6 application I have
>>
>> PYTHONPATH=/nfs/local/lib/python3.6/site-packages
>> PYTHONUSERBASE=/nfs/local
>> PYTHON_VERSION=3.6
>> PYTHON_VIRTUALENV=
>>
>> and for the 3.10 application I have
>>
>>
>> PYTHONPATH=/nfs/easybuild/software/Python/3.10.8-GCCcore-12.2.0/easybuild/python:/nfs/local/lib/python3.10/site-packages
>> PYTHONUSERBASE=/nfs/local
>> PYTHON_VERSION=3.10
>> PYTHON_VIRTUALENV=
>>
>> The applications are installed in /nfs/local/lib/python3.6/site-packages
>> and /nfs/local/lib/python3.10/site-packages, respectively.
>>
>> Can anyone see where this is going wrong? I thought it should be
>> enough that the packages with the metadata is available via PYTHONPATH,
>> but this seems not to be sufficient. So I must be overseeing something.
>
> If in the 3.10 application I add
>
> print(f"__init__ Version: {__version__}")
>
> to __init__.py the correct version is printed. So the problem is that
> the variable is not available at the point I am trying access it. The
> relevant code (a far as I can tell) in main.py looks like this:
>
> import typer
>
> app = typer.Typer()
>
>
> @app.callback()
> def version_callback(value: bool):
> if value:
> typer.echo(f"Version: {__version__}")
> raise typer.Exit()
>
>
> @app.callback()
> def common(
> ctx: typer.Context,
> version: bool = typer.Option(None, "--version",
>help="Show version",
>callback=version_callback),
> ):
> pass
>
> if __name__ == "__main__":
>
> app()
>
> This is the first time I have used typer, so it is more than likely that
> I have made some mistakes.
OK, I worked it out. Instead of
typer.echo(f"Version: {__version__}")
I need
typer.echo(f"Version: {mypackage.__version__}")
Thanks for the help :-)
Even if no-one replies, it still helps me to have to formulate the
problem for an audience of people who probably know more than I do.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
SQL rollback of multiple inserts involving constraints
Hi, In my MariaDB database I have a table 'people' with 'uid' as the primary key and a table 'groups' with 'gid' as the primary key. I have a third table 'memberships' with 'uid' and 'gid' being the primary key and the constraint that values for 'uid' and 'gid' exist in the tables 'people' and 'groups', respectively. I am using SQLAlchemy and writing a method to setup a membership for a new person in a new group. I had assumed that I should be able to perform all three inserts (person, group, membership) with a single transaction and then rollback if there is a problem. However, the problem is that if the both the insert into 'people' and that into 'groups' are not first committed, the constraint on the insertion of the membership fails. What am I doing wrong? Apologies if this is actually an SQL question rather than something related to SQLAlchemy. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: SQL rollback of multiple inserts involving constraints
Jacob Kruger writes: > Think performing a session/transaction flush after the first two > inserts should offer the workaround before you've committed all > transaction actions to the database finally: > > https://medium.com/@oba2311/sqlalchemy-whats-the-difference-between-a-flush-and-commit-baec6c2410a9 > > > HTH Yes, thank you, it does. I hadn't been aware of 'flush'. > Jacob Kruger > +2782 413 4791 > "Resistance is futile!...Acceptance is versatile..." > > > On 2023/11/10 11:15, Loris Bennett via Python-list wrote: >> Hi, >> >> In my MariaDB database I have a table 'people' with 'uid' as the primary >> key and a table 'groups' with 'gid' as the primary key. I have a third >> table 'memberships' with 'uid' and 'gid' being the primary key and the >> constraint that values for 'uid' and 'gid' exist in the tables 'people' >> and 'groups', respectively. I am using SQLAlchemy and writing a method >> to setup a membership for a new person in a new group. >> >> I had assumed that I should be able to perform all three inserts >> (person, group, membership) with a single transaction and then rollback >> if there is a problem. However, the problem is that if the both the >> insert into 'people' and that into 'groups' are not first committed, the >> constraint on the insertion of the membership fails. >> >> What am I doing wrong? >> >> Apologies if this is actually an SQL question rather than something >> related to SQLAlchemy. >> >> Cheers, >> >> Loris >> > -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin -- https://mail.python.org/mailman/listinfo/python-list
Printing dict value for possibly undefined key
Hi,
I want to print some records from a database table where one of the
fields contains a JSON string which is read into a dict. I am doing
something like
print(f"{id} {d['foo']} {d['bar']}")
However, the dict does not always have the same keys, so d['foo'] or
d['bar'] may be undefined. I can obviously do something like
if not 'foo' in d:
d['foo']="NULL"
if not 'bar' in d:
d['bar']="NULL"
print(f"{id} {d['foo']} {d['bar']}")
Is there any more compact way of achieving the same thing?
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Printing dict value for possibly undefined key
duncan smith writes:
> On 24/11/2023 16:35, duncan smith wrote:
>> On 24/11/2023 14:31, Loris Bennett wrote:
>>> Hi,
>>>
>>> I want to print some records from a database table where one of the
>>> fields contains a JSON string which is read into a dict. I am doing
>>> something like
>>>
>>> print(f"{id} {d['foo']} {d['bar']}")
>>>
>>> However, the dict does not always have the same keys, so d['foo'] or
>>> d['bar'] may be undefined. I can obviously do something like
>>>
>>> if not 'foo' in d:
>>> d['foo']="NULL"
>>> if not 'bar' in d:
>>> d['bar']="NULL"
>>> print(f"{id} {d['foo']} {d['bar']}")
>>>
>>> Is there any more compact way of achieving the same thing?
>>>
>>> Cheers,
>>>
>>> Loris
>>>
>> Yes. e.g.
>> d.get('foo', "NULL")
>> Duncan
>
> Or make d a defaultdict.
>
> from collections import defaultdict
>
> dic = defaultdict(lambda:'NULL')
> dic['foo'] = 'astring'
> dic['foo']
> 'astring'
> dic['bar']
> 'NULL'
>
> Duncan
>
I have gone with the 'd.get' solution, as I am just need to print the
dict to the terminal. The dict is actually from a list of dicts which
is generated by querying a database, so I don't think the defaultdict
approach would be so appropriate, but it's good to know about it.
Thanks,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Printing dict value for possibly undefined key
DL Neil writes:
> On 11/25/2023 3:31 AM, Loris Bennett via Python-list wrote:
>> Hi,
>> I want to print some records from a database table where one of the
>> fields contains a JSON string which is read into a dict. I am doing
>> something like
>>print(f"{id} {d['foo']} {d['bar']}")
>> However, the dict does not always have the same keys, so d['foo'] or
>> d['bar'] may be undefined. I can obviously do something like
>>if not 'foo' in d:
>> d['foo']="NULL"
>>if not 'bar' in d:
>> d['bar']="NULL"
>>print(f"{id} {d['foo']} {d['bar']}")
>> Is there any more compact way of achieving the same thing?
>
>
> What does "the dict does not always have the same keys" mean?
>
> a) there are two (or...) keys, but some records don't include both;
>
> b) there may be keys other than 'foo' and 'bar' which not-known in-advance;
>
> c) something else.
Sorry for being unclear. There is either 'foo' or 'bar' or both, plus
some other keys which are always present.
> As mentioned, dict.get() solves one of these.
>
> Otherwise, there are dict methods which collect/reveal all the keys,
> all the values, or both - dict.keys(), .values(), .items(), resp.
That is a also a good point. I had forgotten about dict.keys() and
dict.values(), and hadn't been aware of dict.items().
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Accessing configuration across multiple modules
Hi, I am using Typer to create a command-line program with multiple levels of subcommands, so a typical call might look like mytool --config-file=~/test/mytool.conf serviceXYZ list people In the top-level mytool.main, I evaluate the option '--config-file' and read the config file to initialize the logging. This works fine. However, in the module which lists people, namely mytool.serviceXYZ.cli_people I need to set up a connection to an LDAP server in order to actually read the data. If the LDAP connection details are also in the config file, what is the best way of making them accessible at the point where the object wrapping the LDAP server is initialized? I found this a suggestion here which involves creating a separate module for the configuration and then importing it https://codereview.stackexchange.com/questions/269550/python-share-global-variables-across-modules-from-user-defined-config-file I think I could probably get that to work, but are there any better alternatives? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Configuring an object via a dictionary
Hi, I am initialising an object via the following: def __init__(self, config): self.connection = None self.source_name = config['source_name'] self.server_host = config['server_host'] self.server_port = config['server_port'] self.user_base = config['user_base'] self.user_identifier = config['user_identifier'] self.group_base = config['group_base'] self.group_identifier = config['group_identifier'] self.owner_base = config['owner_base'] However, some entries in the configuration might be missing. What is the best way of dealing with this? I could of course simply test each element of the dictionary before trying to use. I could also just write self.config = config but then addressing the elements will add more clutter to the code. However, with a view to asking forgiveness rather than permission, is there some simple way just to assign the dictionary elements which do in fact exist to self-variables? Or should I be doing this completely differently? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Configuring an object via a dictionary
Tobiah writes: > I should mention that I wanted to answer your question, > but I wouldn't actually do this. I'd rather opt for > your self.config = config solution. The config options > should have their own namespace. > > I don't mind at all referencing foo.config['option'], > or you could make foo.config an object by itself so > you can do foo.config.option. You'd fill it's attributes > in the same way I suggested for your main object. Thanks for the thoughts. I'll go for self.config = config after all, since, as you say, the clutter caused by the referencing is not that significant. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Popping key causes dict derived from object to revert to object
Hi,
I am using SQLAlchemy to extract some rows from a table of 'events'.
>From the call to the DB I get a list of objects of the type
sqlalchemy.orm.state.InstanceState
I would like to print these rows to the terminal using the 'tabulate'
package, the documentation for which says
The module provides just one function, tabulate, which takes a list of
lists or another tabular data type as the first argument, and outputs
a nicely formatted plain-text table
So as I understand it, I need to convert the InstanceState-objects to,
say, dicts, in order to print them. However I also want to remove one
of the keys from the output and assumed I could just pop it off each
event dict, thus:
event_dicts = [vars(e) for e in events]
print(type(event_dicts[0]))
event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts]
print(type(event_dicts[0]))
However, this prints
If I comment out the third line, which pops the unwanted key, I get
Why does popping one of the keys cause the elements of the list to
revert back to their original class?
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Popping key causes dict derived from object to revert to object
Mark Bourne writes:
> Loris Bennett wrote:
>> Hi,
>> I am using SQLAlchemy to extract some rows from a table of 'events'.
>> From the call to the DB I get a list of objects of the type
>>sqlalchemy.orm.state.InstanceState
>> I would like to print these rows to the terminal using the
>> 'tabulate'
>> package, the documentation for which says
>>The module provides just one function, tabulate, which takes a
>> list of
>>lists or another tabular data type as the first argument, and outputs
>>a nicely formatted plain-text table
>> So as I understand it, I need to convert the InstanceState-objects
>> to,
>> say, dicts, in order to print them. However I also want to remove one
>> of the keys from the output and assumed I could just pop it off each
>> event dict, thus:
>>event_dicts = [vars(e) for e in events]
>> print(type(event_dicts[0]))
>> event_dicts = [e.pop('_sa_instance_state', None) for e in event_dicts]
>> print(type(event_dicts[0]))
>
> vars() returns the __dict__ attribute of the object. It may not be a
> good idea to modify that dictionary directly (it will also affect the
> object), although it might be OK if you're not going to do anything
> else with the original objects. To be safer, you could copy the event
> objects:
> event_dicts = [dict(vars(e)) for e in events]
> or:
> event_dicts = [vars(e).copy()]
Thanks for making this clear to me. However, in the end I actually
decided to use the list comprehension without either 'dict()' or
'vars(). Instead I just select the keys I want and so don't need to pop
the unwanted key later and can simultaneously tweak the names of the
key for better printing to the terminal.
>> However, this prints
>>
>>
>> If I comment out the third line, which pops the unwanted key, I get
>>
>>
>> Why does popping one of the keys cause the elements of the list to
>> revert back to their original class?
>
> As Dieter pointed out, the main problem here is that pop() returns the
> value removed, not the dictionary with the rest of the values. You
> probably want something more like:
> for e in event_dicts:
> del e['_sa_instance_state']
> (There's not really any point popping the value if you're not going to
> do anything with it - just delete the key from the dictionary)
Yes, I was mistakenly thinking that the popping the element would leave
me with the dict minus the popped key-value pair. Seem like there is no
such function.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Popping key causes dict derived from object to revert to object
writes:
> Loris wrote:
>
> "Yes, I was mistakenly thinking that the popping the element would leave
> me with the dict minus the popped key-value pair. Seem like there is no
> such function."
>
> Others have tried to explain and pointed out you can del and then use the
> changed dict.
>
> But consider the odd concept of writing your own trivial function.
>
> def remaining(adict, anitem):
> _ = adict.pop(anitem)
> # alternatively duse del on dict and item
> return adict
>
>
remaining({"first": 1, "second": 2, "third": 3}, "second")
> {'first': 1, 'third': 3}
>
>
> Or do you want to be able to call it as in dict.remaining(key) by
> subclassing your own variant of dict and adding a similar method?
No, 'del' does indeed do what I wanted, although I have now decided I
want something else :-) Nevertheless it is good to know that 'del'
exists, so that I don't have to reinvent it.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Popping key causes dict derived from object to revert to object
Grant Edwards writes:
> On 2024-03-22, Loris Bennett via Python-list wrote:
>
>> Yes, I was mistakenly thinking that the popping the element would
>> leave me with the dict minus the popped key-value pair.
>
> It does.
Indeed, but I was thinking in the context of
dict_list = [d.pop('a') for d in dict_list]
and incorrectly expecting to get a list of 'd' without key 'a', instead
of a list of the 'd['a]'.
>> Seem like there is no such function.
>
> Yes, there is. You can do that with either pop or del:
>
> >>> d = {'a':1, 'b':2, 'c':3}
> >>> d
> {'a': 1, 'b': 2, 'c': 3}
> >>> d.pop('b')
> 2
> >>> d
> {'a': 1, 'c': 3}
>
>
> >>> d = {'a':1, 'b':2, 'c':3}
> >>> del d['b']
> >>> d
> {'a': 1, 'c': 3}
>
> In both cases, you're left with the dict minus the key/value pair.
>
> In the first case, the deleted value printed by the REPL because it
> was returned by the expression "d.pop('b')" (a method call).
>
> In the second case is no value shown by the REPL because "del d['b']"
> is a statement not an expression.
Thanks for pointing out 'del'. My main problem, however, was failing to
realise that the list comprehension is populated by the return value of
the 'pop', not the popped dict.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Popping key causes dict derived from object to revert to object
"Michael F. Stemper" writes:
> On 25/03/2024 01.56, Loris Bennett wrote:
>> Grant Edwards writes:
>>
>>> On 2024-03-22, Loris Bennett via Python-list wrote:
>>>
>>>> Yes, I was mistakenly thinking that the popping the element would
>>>> leave me with the dict minus the popped key-value pair.
>>>
>>> It does.
>> Indeed, but I was thinking in the context of
>>dict_list = [d.pop('a') for d in dict_list]
>> and incorrectly expecting to get a list of 'd' without key 'a',
>> instead
>> of a list of the 'd['a]'.
> I apologize if this has already been mentioned in this thread, but are
> you aware of "d.keys()" and "d.values"?
>
> >>> d = {}
> >>> d['do'] = 'a deer, a female deer'
> >>> d['re'] = 'a drop of golden sunshine'
> >>> d['mi'] = 'a name I call myself'
> >>> d['fa'] = 'a long, long way to run'
> >>> d.keys()
> ['fa', 'mi', 'do', 're']
> >>> d.values()
> ['a long, long way to run', 'a name I call myself', 'a deer, a female deer',
> 'a drop of golden sunshine']
> >>>
Yes, I am, thank you. However, I didn't want either the keys or the
values. Instead I wanted to remove a key within a list comprehension.
Cheers,
Loris
PS: "a drop of golden *sun*" - rhymes with "a long, long way to run"
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Dialogs
[email protected] (Stefan Ram) writes: > Me (indented by 2) and the chatbot (flush left). Lines lengths > 72! Is there a name for this kind of indentation, i.e. the stuff you are writing not being flush left? It is sort of contrary to what I think of as "normal" indentation. You seem to use it in all your postings, too, which hurts my brain, but I guess that's my problem :-) [snip (40 lines)] -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Using 'with open(...) as ...' together with configparser.ConfigParser.read
Hi, With Python 3.9.18, if I do try: with open(args.config_file, 'r') as config_file: config = configparser.ConfigParser() config.read(config_file) print(config.sections()) i.e try to read the configuration with the variable defined via 'with ... as', I get [] whereas if I use the file name directly try: with open(args.config_file, 'r') as config_file: config = configparser.ConfigParser() config.read(args.config_file) print(config.sections()) I get ['loggers', 'handlers', 'formatters', 'logger_root', 'handler_fileHandler', 'handler_consoleHandler', 'formatter_defaultFormatter'] which is what I expect. If I print type of 'config_file' I get whereas 'args.config_file' is just Should I be able to use the '_io.TextIOWrapper' object variable here? If so how? Here https://docs.python.org/3.9/library/configparser.html there are examples which use the 'with open ... as' variable for writing a configuration file, but not for reading one. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
Cameron Simpson writes:
> On 01Nov2024 10:10, Loris Bennett wrote:
>>as expected. The non-UTF-8 text occurs when I do
>>
>> mail = EmailMessage()
>> mail.set_content(body, cte="quoted-printable")
>> ...
>>
>> if args.verbose:
>> print(mail)
>>
>>which is presumably also correct.
>>
>>The question is: What conversion is necessary in order to print the
>>EmailMessage object to the terminal, such that the quoted-printable
>>parts are turned (back) into UTF-8?
>
> Do you still have access to `body` ? That would be the original
> message text? Otherwise maybe:
>
> print(mail.get_content())
>
> The objective is to obtain the message body Unicode text (i.e. a
> regular Python string with the original text, unencoded). And to print
> that.
With the following:
##
import email.message
m = email.message.EmailMessage()
m['Subject'] = 'Übung'
m.set_content('Dies ist eine Übung')
print('== cte: default == \n')
print(m)
print('-- full mail ---')
print(m)
print('-- just content--')
print(m.get_content())
m.set_content('Dies ist eine Übung', cte='quoted-printable')
print('== cte: quoted-printable ==\n')
print('-- full mail --')
print(m)
print('-- just content --')
print(m.get_content())
##
I get the following output:
##
== cte: default ==
Subject: Übung
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
RGllcyBpc3QgZWluZSDDnGJ1bmcK
-- full mail ---
Subject: Übung
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
RGllcyBpc3QgZWluZSDDnGJ1bmcK
-- just content--
Dies ist eine Übung
== cte: quoted-printable ==
-- full mail --
Subject: Übung
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Dies ist eine =C3=9Cbung
-- just content --
Dies ist eine Übung
##
So in both cases the subject is fine, but it is unclear to me how to
print the body. Or rather, I know how to print the body OK, but I don't
know how to print the headers separately - there seems to be nothing
like 'get_headers()'. I can use 'get('Subject) etc. and reconstruct the
headers, but that seems a little clunky.
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
Inada Naoki writes: > 2024年11月2日(土) 0:36 Loris Bennett via Python-list : > >> Left Right writes: >> >> > There's quite a lot of misuse of terminology around terminal / console >> > / shell. Please, correct me if I'm wrong, but it looks like you are >> > printing that on MS Windows, right? MS Windows doesn't have or use >> > terminals (that's more of a Unix-related concept). And, by "terminal" >> > I mean terminal emulator (i.e. a program that emulates the behavior of >> > a physical terminal). You can, of course, find some terminal programs >> > for windows (eg. mintty), but I doubt that that's what you are dealing >> > with. >> > >> > What MS Windows users usually end up using is the console. If you >> > run, eg. cmd.exe, it will create a process that displays a graphical >> > console. The console uses an encoding scheme to represent the text >> > output. I believe that the default on MS Windows is to use some >> > single-byte encoding. This answer from SE family site tells you how to >> > set the console encoding to UTF-8 permanently: >> > >> https://superuser.com/questions/269818/change-default-code-page-of-windows-console-to-utf-8 >> > , which, I believe, will solve your problem with how the text is >> > displayed. >> >> I'm not using MS Windows. I am using a Gnome terminal on Debian 12 >> locally and connecting via SSH to a AlmaLinux 8 server, where I start a >> tmux session. >> >> > On Thu, Oct 31, 2024 at 5:19 PM Loris Bennett via Python-list >> > wrote: >> >> >> >> Hi, >> >> >> >> I have a command-line program which creates an email containing German >> >> umlauts. On receiving the mail, my mail client displays the subject and >> >> body correctly: >> >> >> >> Subject: Übung >> >> >> >> Sehr geehrter Herr Dr. Bennett, >> >> >> >> Dies ist eine Übung. >> >> >> >> So far, so good. However, when I use the --verbose option to print >> >> the mail to the terminal via >> >> >> >> if args.verbose: >> >> print(mail) >> >> >> >> I get: >> >> >> >> Subject: Übungsbetreff >> >> >> >> Sehr geehrter Herr Dr. Bennett, >> >> >> >> Dies ist eine =C3=9Cbung. >> >> >> >> What do I need to do to prevent the body from getting mangled? >> >> >> >> I seem to remember that I had issues in the past with a Perl version of >> >> a similar program. As far as I recall there was an issue with fact the >> >> greeting is generated by querying a server, whereas the body is being >> >> read from a file, which lead to oddities when the two bits were >> >> concatenated. But that might just have been a Perl thing. >> >> > > Try PYTHONUTF8=1 envver. > This does not seem to affect the way the email body is printed. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
"Loris Bennett" writes:
> "Loris Bennett" writes:
>
>> Cameron Simpson writes:
>>
>>> On 01Nov2024 10:10, Loris Bennett wrote:
as expected. The non-UTF-8 text occurs when I do
mail = EmailMessage()
mail.set_content(body, cte="quoted-printable")
...
if args.verbose:
print(mail)
which is presumably also correct.
The question is: What conversion is necessary in order to print the
EmailMessage object to the terminal, such that the quoted-printable
parts are turned (back) into UTF-8?
>>>
>>> Do you still have access to `body` ? That would be the original
>>> message text? Otherwise maybe:
>>>
>>> print(mail.get_content())
>>>
>>> The objective is to obtain the message body Unicode text (i.e. a
>>> regular Python string with the original text, unencoded). And to print
>>> that.
>>
>> With the following:
>>
>> ##
>>
>> import email.message
>>
>> m = email.message.EmailMessage()
>>
>> m['Subject'] = 'Übung'
>>
>> m.set_content('Dies ist eine Übung')
>> print('== cte: default == \n')
>> print(m)
>>
>> print('-- full mail ---')
>> print(m)
>> print('-- just content--')
>> print(m.get_content())
>>
>> m.set_content('Dies ist eine Übung', cte='quoted-printable')
>> print('== cte: quoted-printable ==\n')
>> print('-- full mail --')
>> print(m)
>> print('-- just content --')
>> print(m.get_content())
>>
>> ##
>>
>> I get the following output:
>>
>> ##
>>
>> == cte: default ==
>>
>> Subject: Übung
>> Content-Type: text/plain; charset="utf-8"
>> Content-Transfer-Encoding: base64
>> MIME-Version: 1.0
>>
>> RGllcyBpc3QgZWluZSDDnGJ1bmcK
>>
>> -- full mail ---
>> Subject: Übung
>> Content-Type: text/plain; charset="utf-8"
>> Content-Transfer-Encoding: base64
>> MIME-Version: 1.0
>>
>> RGllcyBpc3QgZWluZSDDnGJ1bmcK
>>
>> -- just content--
>> Dies ist eine Übung
>>
>> == cte: quoted-printable ==
>>
>> -- full mail --
>> Subject: Übung
>> MIME-Version: 1.0
>> Content-Type: text/plain; charset="utf-8"
>> Content-Transfer-Encoding: quoted-printable
>>
>> Dies ist eine =C3=9Cbung
>>
>> -- just content --
>> Dies ist eine Übung
>>
>> ##
>>
>> So in both cases the subject is fine, but it is unclear to me how to
>> print the body. Or rather, I know how to print the body OK, but I don't
>> know how to print the headers separately - there seems to be nothing
>> like 'get_headers()'. I can use 'get('Subject) etc. and reconstruct the
>> headers, but that seems a little clunky.
>
> Sorry, I am confusing the terminology here. The 'body' seems to be the
> headers plus the 'content'. So I can print the *content* without the
> headers OK, but I can't easily print all the headers separately. If
> just print the body, i.e. headers plus content, the umlauts in the
> content are not resolved.
OK, so I can do:
##
if args.verbose:
for k in mail.keys():
print(f"{k}: {mail.get(k)}")
print('')
print(mail.get_content())
##
prints what I want and is not wildly clunky, but I am a little surprised
that I can't get a string representation of the whole email in one go.
Cheers,
Loris
--
Dr. Loris Bennett (Herr/Mr)
FUB-IT, Freie Universität Berlin
--
https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
"Loris Bennett" writes:
> Cameron Simpson writes:
>
>> On 01Nov2024 10:10, Loris Bennett wrote:
>>>as expected. The non-UTF-8 text occurs when I do
>>>
>>> mail = EmailMessage()
>>> mail.set_content(body, cte="quoted-printable")
>>> ...
>>>
>>> if args.verbose:
>>> print(mail)
>>>
>>>which is presumably also correct.
>>>
>>>The question is: What conversion is necessary in order to print the
>>>EmailMessage object to the terminal, such that the quoted-printable
>>>parts are turned (back) into UTF-8?
>>
>> Do you still have access to `body` ? That would be the original
>> message text? Otherwise maybe:
>>
>> print(mail.get_content())
>>
>> The objective is to obtain the message body Unicode text (i.e. a
>> regular Python string with the original text, unencoded). And to print
>> that.
>
> With the following:
>
> ##
>
> import email.message
>
> m = email.message.EmailMessage()
>
> m['Subject'] = 'Übung'
>
> m.set_content('Dies ist eine Übung')
> print('== cte: default == \n')
> print(m)
>
> print('-- full mail ---')
> print(m)
> print('-- just content--')
> print(m.get_content())
>
> m.set_content('Dies ist eine Übung', cte='quoted-printable')
> print('== cte: quoted-printable ==\n')
> print('-- full mail --')
> print(m)
> print('-- just content --')
> print(m.get_content())
>
> ##
>
> I get the following output:
>
> ##
>
> == cte: default ==
>
> Subject: Übung
> Content-Type: text/plain; charset="utf-8"
> Content-Transfer-Encoding: base64
> MIME-Version: 1.0
>
> RGllcyBpc3QgZWluZSDDnGJ1bmcK
>
> -- full mail ---
> Subject: Übung
> Content-Type: text/plain; charset="utf-8"
> Content-Transfer-Encoding: base64
> MIME-Version: 1.0
>
> RGllcyBpc3QgZWluZSDDnGJ1bmcK
>
> -- just content--
> Dies ist eine Übung
>
> == cte: quoted-printable ==
>
> -- full mail --
> Subject: Übung
> MIME-Version: 1.0
> Content-Type: text/plain; charset="utf-8"
> Content-Transfer-Encoding: quoted-printable
>
> Dies ist eine =C3=9Cbung
>
> -- just content --
> Dies ist eine Übung
>
> ##
>
> So in both cases the subject is fine, but it is unclear to me how to
> print the body. Or rather, I know how to print the body OK, but I don't
> know how to print the headers separately - there seems to be nothing
> like 'get_headers()'. I can use 'get('Subject) etc. and reconstruct the
> headers, but that seems a little clunky.
Sorry, I am confusing the terminology here. The 'body' seems to be the
headers plus the 'content'. So I can print the *content* without the
headers OK, but I can't easily print all the headers separately. If
just print the body, i.e. headers plus content, the umlauts in the
content are not resolved.
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Specifying local dependency with Poetry
Hi, I have installed a package, 'first-package, built using Poetry and installed it as root in /usr/local. The package is on sys.path so root can successfully import it. I have now developed a second package which depends on 'first-package'. I would have expected that I could specify this dependency in pyproj.toml via [tool.poetry.dependencies] python = "^3.6" first-package = "^1.6.0" However, if I do that and then attempt to install the second package, I get the error Could not find a version that satisfies the requirement first-package<2.0.0,>=1.6.0 (from second-package==0.5.0) (from versions: ) No matching distribution found for first-package<2.0.0,>=1.6.0 (from second-package==0.5.0) How should I define the dependency? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Using 'with open(...) as ...' together with configparser.ConfigParser.read
Jon Ribbens writes: > On 2024-10-29, Loris Bennett wrote: >> Hi, >> >> With Python 3.9.18, if I do >> >> try: >> with open(args.config_file, 'r') as config_file: >> config = configparser.ConfigParser() >> config.read(config_file) >> print(config.sections()) >> >> i.e try to read the configuration with the variable defined via 'with >> ... as', I get >> >>[] >> >> whereas if I use the file name directly >> >> try: >> with open(args.config_file, 'r') as config_file: >> config = configparser.ConfigParser() >> config.read(args.config_file) >> print(config.sections()) >> I get >> >> ['loggers', 'handlers', 'formatters', 'logger_root', >> 'handler_fileHandler', 'handler_consoleHandler', >> 'formatter_defaultFormatter'] >> >> which is what I expect. >> >> If I print type of 'config_file' I get >> >> >> >> whereas 'args.config_file' is just >> >> >> >> Should I be able to use the '_io.TextIOWrapper' object variable here? If so >> how? >> >> Here >> >> https://docs.python.org/3.9/library/configparser.html >> >> there are examples which use the 'with open ... as' variable for writing >> a configuration file, but not for reading one. > > As per the docs you link to, the read() method only takes filename(s) > as arguments, if you have an already-open file you want to read then > you should use the read_file() method instead. As you and others have pointed out, this is indeed covered in the docs, so mea culpa. However, whereas I can see why you might want to read the config from a dict or a string, what would be a use case in which I would want to read from an open file rather than just reading from a file(name)? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Using 'with open(...) as ...' together with configparser.ConfigParser.read
Jon Ribbens writes: > On 2024-10-30, Loris Bennett wrote: >> Jon Ribbens writes: >>> As per the docs you link to, the read() method only takes filename(s) >>> as arguments, if you have an already-open file you want to read then >>> you should use the read_file() method instead. >> >> As you and others have pointed out, this is indeed covered in the docs, >> so mea culpa. >> >> However, whereas I can see why you might want to read the config from a >> dict or a string, what would be a use case in which I would want to >> read from an open file rather than just reading from a file(name)? > > The ConfigParser module provides read(), read_file(), read_string(), > and read_dict() methods. I think they were just trying to be > comprehensive. It's a bit non-Pythonic really. OK, but is there a common situation might I be obliged to use 'read_file'? I.e. is there some common case where the file name is not available, only a corresponding file-like object or stream? -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
"Loris Bennett" writes:
> Cameron Simpson writes:
>
>> On 31Oct2024 16:33, Loris Bennett wrote:
>>>I have a command-line program which creates an email containing German
>>>umlauts. On receiving the mail, my mail client displays the subject and
>>>body correctly:
>> [...]
>>>So far, so good. However, when I use the --verbose option to print
>>>the mail to the terminal via
>>>
>>> if args.verbose:
>>> print(mail)
>>>
>>>I get:
>>>
>>> Subject: Übungsbetreff
>>>
>>> Sehr geehrter Herr Dr. Bennett,
>>>
>>> Dies ist eine =C3=9Cbung.
>>>
>>>What do I need to do to prevent the body from getting mangled?
>>
>> That looks to me like quoted-printable. This is an encoding for binary
>> transport of text to make it robust against not 8-buit clean
>> transports. So your Unicode text is encodings as UTF-8, and then that
>> is encoded in quoted-printable for transport through the email system.
>
> As I mentioned, I think the problem is to do with the way the salutation
> text provided by the "salutation server" and the mail body from a file
> are encoded. This seems to be different.
>
>> Your terminal probably accepts UTF-8 - I imagine other German text
>> renders corectly?
>
> Yes, it does.
>
>> You need to get the text and undo the quoted-printable encoding.
>>
>> If you're using the Python email module to parse (or construct) the
>> message as a `Message` object I'd expect that to happen automatically.
>
> I am using
>
> email.message.EmailMessage
>
> as, from the Python documentation
>
> https://docs.python.org/3/library/email.examples.html
>
> I gathered that that is the standard approach.
>
> And you are right that encoding for the actual mail which is received is
> automatically sorted out. If I display the raw email in my client I get
> the following:
>
> Content-Type: text/plain; charset="utf-8"
> Content-Transfer-Encoding: quoted-printable
> ...
> Subject: =?utf-8?q?=C3=9Cbungsbetreff?=
> ...
> Dies ist eine =C3=9Cbung.
>
> I would interpret that as meaning that the subject and body are encoded
> in the same way.
>
> The problem just occurs with the unsent string representation printed to
> the terminal.
If I log the body like this
body = f"{salutation},\n\n{text}\n{signature}"
logger.debug("body: " + body)
and look at the log file in my terminal I see
2024-11-01 09:59:12,318 - DEBUG - mailer:create_body - body: Sehr geehrter
Herr Dr. Bennett,
Dies ist eine Übung.
...
as expected. The non-UTF-8 text occurs when I do
mail = EmailMessage()
mail.set_content(body, cte="quoted-printable")
...
if args.verbose:
print(mail)
which is presumably also correct.
The question is: What conversion is necessary in order to print the
EmailMessage object to the terminal, such that the quoted-printable
parts are turned (back) into UTF-8?
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
Left Right writes: > There's quite a lot of misuse of terminology around terminal / console > / shell. Please, correct me if I'm wrong, but it looks like you are > printing that on MS Windows, right? MS Windows doesn't have or use > terminals (that's more of a Unix-related concept). And, by "terminal" > I mean terminal emulator (i.e. a program that emulates the behavior of > a physical terminal). You can, of course, find some terminal programs > for windows (eg. mintty), but I doubt that that's what you are dealing > with. > > What MS Windows users usually end up using is the console. If you > run, eg. cmd.exe, it will create a process that displays a graphical > console. The console uses an encoding scheme to represent the text > output. I believe that the default on MS Windows is to use some > single-byte encoding. This answer from SE family site tells you how to > set the console encoding to UTF-8 permanently: > https://superuser.com/questions/269818/change-default-code-page-of-windows-console-to-utf-8 > , which, I believe, will solve your problem with how the text is > displayed. I'm not using MS Windows. I am using a Gnome terminal on Debian 12 locally and connecting via SSH to a AlmaLinux 8 server, where I start a tmux session. > On Thu, Oct 31, 2024 at 5:19 PM Loris Bennett via Python-list > wrote: >> >> Hi, >> >> I have a command-line program which creates an email containing German >> umlauts. On receiving the mail, my mail client displays the subject and >> body correctly: >> >> Subject: Übung >> >> Sehr geehrter Herr Dr. Bennett, >> >> Dies ist eine Übung. >> >> So far, so good. However, when I use the --verbose option to print >> the mail to the terminal via >> >> if args.verbose: >> print(mail) >> >> I get: >> >> Subject: Übungsbetreff >> >> Sehr geehrter Herr Dr. Bennett, >> >> Dies ist eine =C3=9Cbung. >> >> What do I need to do to prevent the body from getting mangled? >> >> I seem to remember that I had issues in the past with a Perl version of >> a similar program. As far as I recall there was an issue with fact the >> greeting is generated by querying a server, whereas the body is being >> read from a file, which lead to oddities when the two bits were >> concatenated. But that might just have been a Perl thing. >> >> Cheers, >> >> Loris >> >> -- >> This signature is currently under constuction. >> -- >> https://mail.python.org/mailman/listinfo/python-list -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin -- https://mail.python.org/mailman/listinfo/python-list
Re: Printing UTF-8 mail to terminal
Cameron Simpson writes: > On 31Oct2024 16:33, Loris Bennett wrote: >>I have a command-line program which creates an email containing German >>umlauts. On receiving the mail, my mail client displays the subject and >>body correctly: > [...] >>So far, so good. However, when I use the --verbose option to print >>the mail to the terminal via >> >> if args.verbose: >> print(mail) >> >>I get: >> >> Subject: Übungsbetreff >> >> Sehr geehrter Herr Dr. Bennett, >> >> Dies ist eine =C3=9Cbung. >> >>What do I need to do to prevent the body from getting mangled? > > That looks to me like quoted-printable. This is an encoding for binary > transport of text to make it robust against not 8-buit clean > transports. So your Unicode text is encodings as UTF-8, and then that > is encoded in quoted-printable for transport through the email system. As I mentioned, I think the problem is to do with the way the salutation text provided by the "salutation server" and the mail body from a file are encoded. This seems to be different. > Your terminal probably accepts UTF-8 - I imagine other German text > renders corectly? Yes, it does. > You need to get the text and undo the quoted-printable encoding. > > If you're using the Python email module to parse (or construct) the > message as a `Message` object I'd expect that to happen automatically. I am using email.message.EmailMessage as, from the Python documentation https://docs.python.org/3/library/email.examples.html I gathered that that is the standard approach. And you are right that encoding for the actual mail which is received is automatically sorted out. If I display the raw email in my client I get the following: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable ... Subject: =?utf-8?q?=C3=9Cbungsbetreff?= ... Dies ist eine =C3=9Cbung. I would interpret that as meaning that the subject and body are encoded in the same way. The problem just occurs with the unsent string representation printed to the terminal. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Seeking Assistance with Python's IDLE for Blind Users
Dear Jeff, writes: > Dear Python Users Group, > > > > I am currently learning Python. I am blind and use the JAWS screen reader to > assist me. I am trying to use Python's IDLE editor but find it quite > challenging. When I move my cursor to a line of code, it reads out the > letters or words from the line above, which makes it very difficult to edit > the code accurately. > > > > I am reaching out to see if anyone knows how to configure IDLE to call out > the letters or words that the cursor is actually on. Additionally, I have > searched extensively online but have not been able to find any documentation > specifically designed for blind users on how to effectively use Python's > IDLE. > > > > Any guidance or resources you could provide would be greatly appreciated. > > > > Thank you for your assistance. > > > > Best regards, > > Jeff > I can't make any suggestions regarding IDLE as I don't use it. I also have the impression that most people, rather than IDLE, use some other form of IDE or editor. So it might be worth looking for more generally programming environments with appropriate support. I personally use Emacs for many activities, including programming in Python. On various Emacs-related mailing lists I have come across blind and visually impaired people, most of whom I assume are using Emacsspeak https://emacspeak.sourceforge.net/ which has been around for 30 years, so that may be worth having a look at. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
FileNotFoundError thrown due to file name in file, rather than file itself
Hi,
I have the following in my program:
try:
logging.config.fileConfig(args.config_file)
config = configparser.ConfigParser()
config.read(args.config_file)
if args.verbose:
print(f"Configuration file: {args.config_file}")
except FileNotFoundError:
print(f"Error: configuration file {args.config_file} not found.
Exiting.")
sys.exit(0)
and when I ran the program I got the error
Error: configuration file /usr/local/etc/sc_mailer not found. Exiting.
However, this file *does* exist and *can* be read. By checking the
'filename' attribute of the exception I discovered that the problem was
the log file defined *in* the config file, namely
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=defaultFormatter
args=('/var/log/my_prog.log', 'a')
This log file did not exist. The exception is thrown by
logging.config.fileConfig(args.config_file)
My questions are:
1. Should I be surprised by this behaviour?
2. In terms of generating a helpful error message, how should one
distinguish between the config file not existing and the log file not
existing?
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: FileNotFoundError thrown due to file name in file, rather than file itself
Chris Angelico writes: > On Tue, 12 Nov 2024 at 01:59, Loris Bennett via Python-list > wrote: >> 2. In terms of generating a helpful error message, how should one >>distinguish between the config file not existing and the log file not >>existing? > > By looking at the exception's attributes rather than assuming and > hard-coding the path in your message? Or, even better, just let the > exception bubble. I didn't consider letting the exception bubble as is the top of the code for a CLI program. I was hoping to just catch what I thought might be a common error of the config file being missing. -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: FileNotFoundError thrown due to file name in file, rather than file itself
Cameron Simpson writes: > On 11Nov2024 18:24, [email protected] wrote: >>Loris Bennett wrote at 2024-11-11 15:05 +0100: >>>I have the following in my program: >>>try: >>>logging.config.fileConfig(args.config_file) >>>config = configparser.ConfigParser() >>>config.read(args.config_file) >>>if args.verbose: >>>print(f"Configuration file: {args.config_file}") >>>except FileNotFoundError: >>>print(f"Error: configuration file {args.config_file} not found. >>> Exiting.") >> >>Do not replace full error information (including a traceback) >>with your own reduced error message. >>If you omit your "try ... except FileNotFoundError` >>(or start the `except` clause with a `raise`), you >>will learn where in the code the exception has been raised >>and likely as well what was not found (Python is quite good >>with such error details). > > Actually, file-not-found is pretty well defined - the except action > itself is fine in that regard. > > [...] >>>2. In terms of generating a helpful error message, how should one >>> distinguish between the config file not existing and the log file not >>> existing? > > Generally you should put a try/except around the smallest possible > piece of code. So: > > config = configparser.ConfigParser() > try: > config.read(args.config_file) > except FileNotFoundError as e: > print(f"Error: configuration file {args.config_file} not found: {e}") > > This way you know that the config file was missing. I appreciate the point you make about the smallest possible piece of code, although I can imagine that this could potentially create a lot of try/except clutter and I might just want a single block and then try to catch various exceptions. Regarding your example above, if 'missingfile.py' contains the following import configparser config = configparser.ConfigParser() try: config.read('/foo/bar') except FileNotFoundError as e: print(f"Error: configuration file {config_file} not found: {e}") them python3 missingfile.py does not produce an any output for me and so does not seem to be a reliable way of handling the case where the config file does not exist. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: FileNotFoundError thrown due to file name in file, rather than file itself
Chris Angelico writes: > On Tue, 12 Nov 2024 at 01:59, Loris Bennett via Python-list > wrote: >> 2. In terms of generating a helpful error message, how should one >>distinguish between the config file not existing and the log file not >>existing? > > By looking at the exception's attributes rather than assuming and > hard-coding the path in your message? Or, even better, just let the > exception bubble. As Dieter also pointed out, I obviously made a mistake in assuming that I knew what 'FileNotFound' was referring to. However, it strikes me as not immediately obvious that the logging file must exist at this point. I can imagine a situation in which I want to configure a default log file and create it if it missing. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: FileNotFoundError thrown due to file name in file, rather than file itself
Left Right writes:
> Poor error reporting is a very common problem in programming. Python
> is not anything special in this case. Of course, it would've been
> better if the error reported what file wasn't found. But, usually
> these problems are stacking, like in your code. Unfortunately, it's
> your duty, as the language user, to anticipate those problems and act
> accordingly. Now you've learned that the one file you believe that
> could be the source for the error isn't the only one--well, adjust
> your code to differentiate between those two (and potentially other?)
> cases. There's very little else you can do beside that.
>
> NB. On the system level, the error has no information about what file
> wasn't found. It simply returns some numeric value (the famous
> ENOENT) in case when the system call to open a file fails. Python
> could've been more helpful by figuring out what path caused the
> problem and printing that in the error message, but it doesn't...
> That's why I, myself, never use the vanilla FileNotFoundError, I
> always re-rise it with a customized version that incorporates the
> information about the missing file in the error message.
That sounds like a good idea.
> NB2. It's always a bad idea to print logs to files. Any sysadmin /
> ops / infra person worth their salt will tell you that. The only
> place the logs should go to is the standard error. There are true and
> tried tools that can pick up logs from that point on, and do with them
> whatever your heart desires. That is, of course, unless you are
> creating system tools for universal log management (in which case, I'd
> question the choice of Python as a suitable language for such a task).
> Unfortunately, even though this has been common knowledge for decades,
> it's still elusive in the world of application development :|
I am not entirely convinced by NB2. I am, in fact, a sort of sysadmin
person and most of my programs write to a log file. The programs are
also moderately complex, so a single program might access a database,
query an LDAP server, send email etc., so potentially quite a lot can go
wrong. They are also not programs whose output I would pipe to another
command. What would be the advantage of logging to stderr? Quite apart
from that, I find having a log file a useful for debugging when I am
developing.
Cheers,
Loris
> On Mon, Nov 11, 2024 at 4:00 PM Loris Bennett via Python-list
> wrote:
>>
>> Hi,
>>
>> I have the following in my program:
>>
>> try:
>> logging.config.fileConfig(args.config_file)
>> config = configparser.ConfigParser()
>> config.read(args.config_file)
>> if args.verbose:
>> print(f"Configuration file: {args.config_file}")
>> except FileNotFoundError:
>> print(f"Error: configuration file {args.config_file} not found.
>> Exiting.")
>> sys.exit(0)
>>
>> and when I ran the program I got the error
>>
>> Error: configuration file /usr/local/etc/sc_mailer not found. Exiting.
>>
>> However, this file *does* exist and *can* be read. By checking the
>> 'filename' attribute of the exception I discovered that the problem was
>> the log file defined *in* the config file, namely
>>
>> [handler_fileHandler]
>> class=FileHandler
>> level=DEBUG
>> formatter=defaultFormatter
>> args=('/var/log/my_prog.log', 'a')
>>
>> This log file did not exist. The exception is thrown by
>>
>> logging.config.fileConfig(args.config_file)
>>
>> My questions are:
>>
>> 1. Should I be surprised by this behaviour?
>> 2. In terms of generating a helpful error message, how should one
>>distinguish between the config file not existing and the log file not
>>existing?
>>
>> Cheers,
>>
>> Loris
>>
>> --
>> This signature is currently under constuction.
>> --
>> https://mail.python.org/mailman/listinfo/python-list
--
Dr. Loris Bennett (Herr/Mr)
FUB-IT, Freie Universität Berlin
--
https://mail.python.org/mailman/listinfo/python-list
Printing UTF-8 mail to terminal
Hi, I have a command-line program which creates an email containing German umlauts. On receiving the mail, my mail client displays the subject and body correctly: Subject: Übung Sehr geehrter Herr Dr. Bennett, Dies ist eine Übung. So far, so good. However, when I use the --verbose option to print the mail to the terminal via if args.verbose: print(mail) I get: Subject: Übungsbetreff Sehr geehrter Herr Dr. Bennett, Dies ist eine =C3=9Cbung. What do I need to do to prevent the body from getting mangled? I seem to remember that I had issues in the past with a Perl version of a similar program. As far as I recall there was an issue with fact the greeting is generated by querying a server, whereas the body is being read from a file, which lead to oddities when the two bits were concatenated. But that might just have been a Perl thing. Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Poetry: endpoints with endpoints
Hi,
I am using Poetry and have the following in my pyproj.toml
[tool.poetry.scripts]
frobnicate = "frobnicator.cli:frobnicate"
The CLI provides an option '--flavour' and I would like to add further
endpoints for specific values of 'flavour'. I tried adding
frobnicate_foo = "frobnicator.cli:frobnicate --flavour foo"
to '[tool.poetry.scripts]', but when I call this I get the error
$ poetry run frobnicate_foo --verbose
File "", line 1
import sys; from importlib import import_module; sys.argv = ['frobnicate,
'--verbose']; import_module('frobniator.cli').frobnicate --flavour foo()
^
SyntaxError: invalid syntax
Is it possible to add such endpoint? If so, how?
Cheers,
Loris
--
This signature is currently under constuction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Using 'with open(...) as ...' together with configparser.ConfigParser.read
Jon Ribbens writes: > On 2024-10-30, Loris Bennett wrote: >> Jon Ribbens writes: >>> On 2024-10-30, Loris Bennett wrote: Jon Ribbens writes: > As per the docs you link to, the read() method only takes filename(s) > as arguments, if you have an already-open file you want to read then > you should use the read_file() method instead. As you and others have pointed out, this is indeed covered in the docs, so mea culpa. However, whereas I can see why you might want to read the config from a dict or a string, what would be a use case in which I would want to read from an open file rather than just reading from a file(name)? >>> >>> The ConfigParser module provides read(), read_file(), read_string(), >>> and read_dict() methods. I think they were just trying to be >>> comprehensive. It's a bit non-Pythonic really. >> >> OK, but is there a common situation might I be obliged to use >> 'read_file'? I.e. is there some common case where the file name is not >> available, only a corresponding file-like object or stream? > > Well, sure - any time it's not being read from a file. A bit ironic > that the method to use in that situation is "read_file", of course. > In my view the read() and read_file() methods have their names the > wrong way round. But bear in mind this code is 27 years old, and > the read() function came first. Yes, I suppose history has a lot to answer for :-) However I didn't make myself clear: I understand that there are different functions, depending on whether I have a file name or a stream. Nevertheless, I just can't think of a practical example where I might just have *only* a stream, especially one containing my configuration data. I was just interested to know if anyone can give an example. -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Common objects for CLI commands with Typer
Hi, Apologies if the following description is to brief - I can expand if no one knows what I'm on about, but maybe a short description is enough. I am developing a command line application using Typer. Most commands need to do something in a database and also do LDAP stuff. Currently each command creates its own Database and LDAP objects, since each command forms an entry point to the program. With Typer, is there a way I can define the equivalent of class attributes at a single point which are then available to all commands? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: Specifying local dependency with Poetry
Greg Ewing writes: > On 6/11/24 4:13 am, Loris Bennett wrote: >> [tool.poetry.dependencies] >> python = "^3.6" >> first-package = "^1.6.0" >>Could not find a version that satisfies the requirement >> first-package<2.0.0,>=1.6.0 (from second-package==0.5.0) (from >> versions: ) >> No matching distribution found for first-package<2.0.0,>=1.6.0 (from >> second-package==0.5.0) > > What version number does first-package have? > > The caret in the specification "^1.6.0" restricts it to 1.x.x versions. > > If you don't want that restriction, use ">=" instead. The first package has the version number 1.6.0. -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
Re: FileNotFoundError thrown due to file name in file, rather than file itself
Left Right writes: >> I am not entirely convinced by NB2. I am, in fact, a sort of sysadmin >> person and most of my programs write to a log file. The programs are >> also moderately complex, so a single program might access a database, >> query an LDAP server, send email etc., so potentially quite a lot can go >> wrong. They are also not programs whose output I would pipe to another >> command. What would be the advantage of logging to stderr? Quite apart >> from that, I find having a log file a useful for debugging when I am >> developing. > > First, the problem with writing to files is that there is no way to > make these logs reliable. This is what I mean by saying these are > unreliable: since logs are designed to grow indefinitely, the natural > response to this design property is log rotation. But, it's > impossible to reliably rotate a log file. There's always a chance > that during the rotation some log entries will be written to the file > past the point of rotation, but prior to the point where the next logs > volume starts. > There are similar reliability problems with writing to Unix or > Internet sockets, databases etc. For different reasons, but at the > end of the day, whoever wants logs, they want them to be reliable. > Both simplicity and convention selected for stderr as the only and the > best source of logging output. If I understand correctly you are not saying that logrotate is less reliable that the other methods mentioned above. But in that case, reliability seems no more of a reason not to log to files than it is a reason not to write to a socket or to a database. > Programs that write their output to log files will always irritate > their users because users will have to do some detective work to > figure out where those files are, and in some cases they will have to > do administrative works to make sure that the location where the > program wants to store the log files is accessible, has enough free > space, is speedy enough etc. All your points regarding the work involved are valid, but many programs, such as MariaDB, OpenLDAP or SSSD, do write to a log file (and it is usually under /var/log or /var/log/something. So it seems like a common approach. Besides, I define the location of the logfile in the config file for the program (the problem in my original question arose from this filename in the config file not existing). So finding the location is not an issue. You have to find the config file, of course, but I think /etc or /usr/local/etc are fairly standard and my programs generally have an option '--config-file' anyway. > So, from the ops perspective, whenever I > come across a program that tries to write logs to anything other than > stderr, I make an earnest effort to throw that program into the gutter > and never touch it again. It's too much headache to babysit every > such program, to remember the location of the log files of every such > program, the required permissions, to provision storage. If you are > in that line of work, you just want all logs to go to the same place > (journal), where you can later filter / aggregate / correlate and > perform other BI tasks as your heart desires. That may be true in many cases, but those I am dealing with don't require much filtering beyond 'grep' and also don't require aggregation or correlation. > Of course, if you only administer your own computer, and you have low > single digits programs to run, and their behavior doesn't change > frequently, and you don't care to drop some records every now and > then... it's OK to log to files directly from a program. But then you > aren't really in the sysadmin / infra / ops category, as you are more > of a hobby enthusiast. What I do is indeed a bit of a niche, but I do get paid for this, so I would not consider myself a 'hobby enthusiast'. > Finally, if you want your logs to go to a file, and currently, your > only option is stderr, your shell gives you a really, really simple > way of redirecting stderr to a file. So, really, there aren't any > excuses to do that. I don't quite understand what your suggestion is. Do you mean that I should log to stderr and then run my program as my_program ... 2>&1 | logger ? Cheers, Loris -- This signature is currently under constuction. -- https://mail.python.org/mailman/listinfo/python-list
