CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Samuel Marks]

Previously I have solved related problems with explicit `-}` and `-{`
(or `-b`) as in `nginxctl`:
```
$ python -m nginxctl serve --temp_dir '/tmp' \
-b 'server' \
  --server_name 'localhost' --listen '8080' \
  -b location '/' \
--root '/tmp/wwwroot' \
  -} \
-}
nginx is running. Stop with: /usr/local/bin/nginx -c /tmp/nginx.conf -s stop
```

To illustrate the issue, using `ml-params` and ml-params-tensorflow:
```
$ python -m ml_params --engine 'tensorflow' train --help

usage: python -m ml_params train [-h]
 --optimizer

{Adadelta,Adagrad,Adam,Adamax,Ftrl,Nadam,RMSprop}

Run the training loop for your ML pipeline.

optional arguments:
  -h, --helpshow this help message and exit
  --optimizer {Adadelta,Adagrad,Adam,Adamax,Ftrl,Nadam,RMSprop}
The optimizer
```
Idea: preprocess `sys.argv` so that this syntax would work
`--optimizer Adam[learning_rate=0.01]`*

*square rather than round so as not to require escape characters or
quoting in `sh`

Unfortunately this approach wouldn't give nice `--help` text.

What's the right solution here?

Thanks,

Samuel Marks
Charity  | consultancy
 | open-source  |
LinkedIn 
-- 
https://mail.python.org/mailman/listinfo/python-list

Efficient authorization for Django list views with oso

[Stephie Glaser]

Hi All! We're building an open source policy engine for adding access
control (permissions, roles, etc.) to apps, called oso. oso policies are
declarative, and enable users to cleanly separate authorization logic from
the rest of their application code.

In our latest release of django-oso, we added functionality to enforce
authorization policies directly on Django QuerySets. The declarative policy
is translated into filters that can be understood by the Django ORM, in a
way that is fully abstracted from the library user. In the future, we'll be
adding support for SQLAlchemy and other Python ORMs.

We wrote about it in this blog post.



Cleanly separating an authorization policy from other application code can
be challenging, especially when authorizing a collection of objects, often
necessary in list views. This feature makes it possible to enforce
authorization as a filter when querying the application data store while
still taking full advantage of the declarative policy provided by oso.

This functionality is in preview in the 0.3.0 release of django-oso
.

Install via pip: pip install django-oso

Let us know what you think in our Slack channel or on this thread!
join-slack.osohq.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Dieter Maurer]

Samuel Marks wrote at 2020-10-15 20:53 +1100:
> ...
>To illustrate the issue, using `ml-params` and ml-params-tensorflow:
> ...
>What's the right solution here?

While Python provides several modules in its standard library
to process parameters (e.g. the simple `getopt` and the flexible `argparse`),
it is up to the "application" whether it uses such a module (and which one)
or whether it handle arguments on its own.

Apparently, `ml_param` is not a staudard Python module.
Is it a package of your own? Then I suggest to check `argparse` whether
it supports your use case (I know, it can be customized to do it,
but maybe, it does it already out of the box).

If `ml_param` is a third party module, then the question
is actually an `ml_param` question. Ask its support mailing lists
or have a look at its source.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Simple question - end a raw string with a single backslash ?

[Roland Müller via Python-list]

On 10/13/20 4:14 PM, Serhiy Storchaka wrote:

13.10.20 11:52, Tony Flury via Python-list пише:

I am trying to write a simple expression to build a raw string that ends
in a single backslash. My understanding is that a raw string should
ignore attempts at escaping characters but I get this :

     >>> a = r'end\'
       File "", line 1
         a = r'end\'
       ^
    SyntaxError: EOL while scanning string literal

I interpret this as meaning that the \' is actually being interpreted as
a literal quote - is that a bug ?

r'You can\'t end raw string literal with a single "\"'

If backslash be true inner in a raw string, the above literal would end
after \'. It would be very hard to write a raw string containing both \'
and \", and even \''' and \""" (there are such strings in the stdlib).

So you have problem either with trailing backslash, or with inner
backslash followed by quotes. Both problems cannot be solved at the same
time. Python parser works as it works because initially it was easier to
implement, and now this cannot be changed because it would break some
amount of correct code.


The only solution I have found is to do this :

     >>> a = r'end' + chr(92)
     >>> a
    'end\\'
     >>> list(a)
    ['e', 'n', 'd', '\\']

or


     >>> a = r'end\\'[:-1]
     >>> list(a)
    ['e', 'n', 'd', '\\']

Neither of which are nice.

You can also write r'end' '\\'. It is not too nice, but it looks nicer
to me then two other variants.


I used the triple single quotes as delimiter:

>>> s = r'''a single quote ', a double quote "'''
>>> s

'a single quote \', a double quote "'

BR,

Roland


--
https://mail.python.org/mailman/listinfo/python-list

Re: Simple question - end a raw string with a single backslash ?

[Serhiy Storchaka]

15.10.20 22:16, Roland Müller via Python-list пише:
> I used the triple single quotes as delimiter:
> 
 s = r'''a single quote ', a double quote "'''
 s
> 
> 'a single quote \', a double quote "'

It does not help if the string contains both kinds of triple quotes

   You have to use triple quotes (''', """) for docstrings.

or contains one kind of triple quote and ends with a single quote of
other kind.

   a triple single quote ''', a single double quote "

Of course there are workarounds, but the fact that changing the current
rules will break existing code.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Samuel Marks]

Yes it’s my module, and I’ve been using argparse
https://github.com/SamuelMarks/ml-params

No library I’ve found provides a solution to CLI argument parsing for my
use-case.

So I’ll write one. But what should it look like, syntactically and
semantically?

On Fri, 16 Oct 2020 at 3:14 am, Dieter Maurer  wrote:

> Samuel Marks wrote at 2020-10-15 20:53 +1100:
> > ...
> >To illustrate the issue, using `ml-params` and ml-params-tensorflow:
> > ...
> >What's the right solution here?
>
> While Python provides several modules in its standard library
> to process parameters (e.g. the simple `getopt` and the flexible
> `argparse`),
> it is up to the "application" whether it uses such a module (and which one)
> or whether it handle arguments on its own.
>
> Apparently, `ml_param` is not a staudard Python module.
> Is it a package of your own? Then I suggest to check `argparse` whether
> it supports your use case (I know, it can be customized to do it,
> but maybe, it does it already out of the box).
>
> If `ml_param` is a third party module, then the question
> is actually an `ml_param` question. Ask its support mailing lists
> or have a look at its source.
>
> --
Samuel Marks
Charity  | consultancy
 | open-source
 | LinkedIn <
https://linkedin.com/in/samuelmarks>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Cameron Simpson]

On 16Oct2020 10:09, Samuel Marks  wrote:
>Yes it’s my module, and I’ve been using argparse
>https://github.com/SamuelMarks/ml-params
>
>No library I’ve found provides a solution to CLI argument parsing for my
>use-case.
>
>So I’ll write one. But what should it look like, syntactically and
>semantically?

I would make it look like argparse, if you're using argparse already. 
Can argparse be subclassed, eg to override some internal 
option-gathering method?

In particular, I would not invent yet another command line syntax.

I'd stick with the common -a and --long-name style, maybe using the -{ -} 
stuff you cited from nginx if it meets your needs. As with any 
bracketing syntax, you may one day find yourself needing to embed the 
literal end-bracket symbol in your command line one day, and thus may 
need some kind of escaping notion ("no! not closing the brackets yet!").

I can't offer mch specific advice on argparse, I use getopt myself.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Cameron Simpson]

One other thing:

On 15Oct2020 20:53, Samuel Marks  wrote:
>Idea: preprocess `sys.argv` so that this syntax would work
>`--optimizer Adam[learning_rate=0.01]`*
>
>*square rather than round so as not to require escape characters or
>quoting in `sh`

Square brackets are also shell syntax, introducing glob character
ranges. You're only not noticing probably because an unmatched glob is
normally let through unchanged.

For example:

   somecmd x[b]

would go through as "x[b]" _unless_ you had a matching local filename
(in this case a file named "xb") in which case the shell would match the
glob and pass through the match (in this case "xb").

You'll find this issue a fair bit: if you want some punctuation, someone
else has probably already wanted that punctuation for something. You'll
still need to quote things for a lot of stuff.

Commas are not special. You could try:

--optimizer Adam,learning_rate=0.01,something_else=3

which would work for a fair amount of stuff.

>Unfortunately this approach wouldn't give nice `--help` text.
>What's the right solution here?

To get nice help text you need your command line parser to be able to
compose that help text in some way. Usually the help text in argparse is
supplied when you define the option.

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Samuel Marks]

--optimizer Adam,learning_rate=0.01,something_else=3

That syntax isn’t so bad! =]

How would you suggest the help text for this looks? (don’t worry about
implementation, just what goes to stdout/stderr)

PS: Yeah I used square brackets for my Bash arrays

On Fri, 16 Oct 2020 at 10:26 am, Cameron Simpson  wrote:

> One other thing:
>
> On 15Oct2020 20:53, Samuel Marks  wrote:
> >Idea: preprocess `sys.argv` so that this syntax would work
> >`--optimizer Adam[learning_rate=0.01]`*
> >
> >*square rather than round so as not to require escape characters or
> >quoting in `sh`
>
> Square brackets are also shell syntax, introducing glob character
> ranges. You're only not noticing probably because an unmatched glob is
> normally let through unchanged.
>
> For example:
>
>somecmd x[b]
>
> would go through as "x[b]" _unless_ you had a matching local filename
> (in this case a file named "xb") in which case the shell would match the
> glob and pass through the match (in this case "xb").
>
> You'll find this issue a fair bit: if you want some punctuation, someone
> else has probably already wanted that punctuation for something. You'll
> still need to quote things for a lot of stuff.
>
> Commas are not special. You could try:
>
> --optimizer Adam,learning_rate=0.01,something_else=3
>
> which would work for a fair amount of stuff.
>
> >Unfortunately this approach wouldn't give nice `--help` text.
> >What's the right solution here?
>
> To get nice help text you need your command line parser to be able to
> compose that help text in some way. Usually the help text in argparse is
> supplied when you define the option.
>
> Cheers,
> Cameron Simpson 
>
-- 
Samuel Marks
Charity  | consultancy
 | open-source
 | LinkedIn <
https://linkedin.com/in/samuelmarks>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Karen Shaeffer via Python-list]

Hi Sam,
I’ve been using abseil python API.

https://abseil.io/docs/python/guides/flags 

https://abseil.io/docs/python/quickstart 


It’s a distributed command line system with features that appear to support 
your needs.

Karen


> On Oct 15, 2020, at 4:09 PM, Samuel Marks  wrote:
> 
> Yes it’s my module, and I’ve been using argparse
> https://github.com/SamuelMarks/ml-params 
> 
> 
> No library I’ve found provides a solution to CLI argument parsing for my
> use-case.
> 
> So I’ll write one. But what should it look like, syntactically and
> semantically?
> 
> On Fri, 16 Oct 2020 at 3:14 am, Dieter Maurer  > wrote:
> 
>> Samuel Marks wrote at 2020-10-15 20:53 +1100:
>>> ...
>>> To illustrate the issue, using `ml-params` and ml-params-tensorflow:
>>> ...
>>> What's the right solution here?
>> 
>> While Python provides several modules in its standard library
>> to process parameters (e.g. the simple `getopt` and the flexible
>> `argparse`),
>> it is up to the "application" whether it uses such a module (and which one)
>> or whether it handle arguments on its own.
>> 
>> Apparently, `ml_param` is not a staudard Python module.
>> Is it a package of your own? Then I suggest to check `argparse` whether
>> it supports your use case (I know, it can be customized to do it,
>> but maybe, it does it already out of the box).
>> 
>> If `ml_param` is a third party module, then the question
>> is actually an `ml_param` question. Ask its support mailing lists
>> or have a look at its source.
>> 
>> --
> Samuel Marks
> Charity > | 
> consultancy
> > | open-source
> > | LinkedIn <
> https://linkedin.com/in/samuelmarks >
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[2QdxY4RzWzUUiLuE]

On 2020-10-16 at 10:59:16 +1100,
Samuel Marks  wrote:

> --optimizer Adam,learning_rate=0.01,something_else=3
> 
> That syntax isn’t so bad! =]
> 
> How would you suggest the help text for this looks? (don’t worry about
> implementation, just what goes to stdout/stderr)

--optimizer name[,option=value...]

Obviously, you'll have to explain that the brackets and the words option
and value are meta syntax, but you're probably already doing that for
your current meta syntax anyway.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[2QdxY4RzWzUUiLuE]

On 2020-10-16 at 10:20:40 +1100,
Cameron Simpson  wrote:

> On 16Oct2020 10:09, Samuel Marks  wrote:
> >Yes it’s my module, and I’ve been using argparse
> >https://github.com/SamuelMarks/ml-params
> >
> >No library I’ve found provides a solution to CLI argument parsing for my
> >use-case.

Out of curiosity, what do your command line interfaces require that
argparse and other libraries don't have?

Yes, if my only tool is a hammer, then every problem looks like a nail,
but I've yet to have a requirement that the POSIX rules don't cover.

> >So I’ll write one ...

Been there.  Done that.  :-)

> > [...] But what should it look like, syntactically and semantically?

[...]

> In particular, I would not invent yet another command line syntax.

I agree.  The POSIX Utility Syntax and Guidelines:

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html

(I believe argparse supports a superset of the POSIX syntax.)

I haven't found a definitive definition of GNU long options, only
examples.

HTH,
Dan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Samuel Marks]

Hi Dan,

The feature that existing CLI parsers are missing is a clean syntax
for specifying options on the second parameter (the "value"), where
there may be different options available depending on which you
choose.

For example:
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam
has `learning_rate`, `beta_1`, `beta_2`, `epsilon`, and `amsgrad`*
Whereas
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/RMSprop
has `learning_rate`, `rho`, `momentum`, `epsilon`, `centered`*

*with clipnorm, clipvalue, decay hidden behind kwargs

So the question is how to expose this as CLI options. `--optimizer
Adam` is a good first step, but it should error if I try and give it
`momentum`. The comma syntax is my favourite so far.

I guess I'll just have to write a validator outside the CLI parser to
handle this…

Samuel Marks
Charity  | consultancy
 | open-source  |
LinkedIn 

On Fri, Oct 16, 2020 at 11:19 AM <2qdxy4rzwzuui...@potatochowder.com> wrote:
>
> On 2020-10-16 at 10:20:40 +1100,
> Cameron Simpson  wrote:
>
> > On 16Oct2020 10:09, Samuel Marks  wrote:
> > >Yes it’s my module, and I’ve been using argparse
> > >https://github.com/SamuelMarks/ml-params
> > >
> > >No library I’ve found provides a solution to CLI argument parsing for my
> > >use-case.
>
> Out of curiosity, what do your command line interfaces require that
> argparse and other libraries don't have?
>
> Yes, if my only tool is a hammer, then every problem looks like a nail,
> but I've yet to have a requirement that the POSIX rules don't cover.
>
> > >So I’ll write one ...
>
> Been there.  Done that.  :-)
>
> > > [...] But what should it look like, syntactically and semantically?
>
> [...]
>
> > In particular, I would not invent yet another command line syntax.
>
> I agree.  The POSIX Utility Syntax and Guidelines:
>
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html
>
> (I believe argparse supports a superset of the POSIX syntax.)
>
> I haven't found a definitive definition of GNU long options, only
> examples.
>
> HTH,
> Dan
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Cameron Simpson]

On 16Oct2020 10:59, Samuel Marks  wrote:
>--optimizer Adam,learning_rate=0.01,something_else=3
>
>That syntax isn’t so bad! =]
>
>How would you suggest the help text for this looks? (don’t worry about
>implementation, just what goes to stdout/stderr)

Maybe:

Usage: ...
  
  --optimizer name[,param=value...]
Specify use of the named optimizer with optional additional
parameters:
  learning_rate   Scale of gullibility, a float.
  something_else  Another tuning thing, 
  ...

which is the style I use in my own usage messages. Possibly also allow the 
"=value" to be optional (where sensible per parameter), since it is so easy.

>PS: Yeah I used square brackets for my Bash arrays

By the time I want real arrays in my shell scripts I'm often verging on the 
regime where it's time to transition to a more expressive language. Like 
Python :-) (Unless of course you're using a bunch of shell control constructs 
like pipes, which are succintly written in the shell.)

Cheers,
Cameron Simpson 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[2QdxY4RzWzUUiLuE]

On 2020-10-16 at 11:27:56 +1100,
Regarding "Re: CLI parsing—with `--help` text—`--foo bar`, how to give 
additional parameters to `bar`?,"
Samuel Marks  wrote:

> The feature that existing CLI parsers are missing is a clean syntax
> for specifying options on the second parameter (the "value"), where
> there may be different options available depending on which you
> choose.
> 
> For example:
> https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam
> has `learning_rate`, `beta_1`, `beta_2`, `epsilon`, and `amsgrad`*
> Whereas
> https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/RMSprop
> has `learning_rate`, `rho`, `momentum`, `epsilon`, `centered`*
> 
> *with clipnorm, clipvalue, decay hidden behind kwargs
> 
> So the question is how to expose this as CLI options. `--optimizer
> Adam` is a good first step, but it should error if I try and give it
> `momentum`. The comma syntax is my favourite so far.

I guess to me, that level of complexity (an optimizer followed by a
series of key/value pairs) is at least partially an application problem
rather than a library problem.  IOW, if my command line parser tells me
that the user specified "--optimizer" with an argument of
"Adam,learning_rate=1.0,beta_1=3.0," then I'll parse the rest in my
application.  At some point, I might consider a lex/yacc or ini-file
solution instead of that kind of command line.

> I guess I'll just have to write a validator outside the CLI parser to
> handle this…

Exactly.  :-)

-- 
“Whoever undertakes to set himself up as a
judge of Truth and Knowledge is shipwrecked
by the laughter of the gods.” – Albert Einstein
Dan Sommers, http://www.tombstonezero.net/dan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Dieter Maurer]

Samuel Marks wrote at 2020-10-16 10:09 +1100:
>Yes it’s my module, and I’ve been using argparse
>https://github.com/SamuelMarks/ml-params
>
>No library I’ve found provides a solution to CLI argument parsing for my
>use-case.

Do you know that with `argparse` you can specify how many arguments an option
expects? Thus, it should be quite easily possible to
have --opt   ...
Do you know that you can define new `Action`s for `argparse`?
This way, you could properly process `--opt ,, ...`.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?

[Samuel Marks]

Yeah I've played with custom actions before
https://github.com/offscale/argparse-utils/tree/master/argparse_utils/actions

But this would only help in one phase, the important phase of
providing help text will need to be provided out-of-argparse and
thrown in

(like my trivial absl alternative, exposing a function which takes an
argparse instance and returns an argparse instance)

The only hack remaining is that I have to pass through `sys.argv` at
least once before giving it to argparse. I wonder if there's a way to
not need to explicitly go through it at all…
https://github.com/SamuelMarks/ml-params/blob/d1fb184/ml_params/__main__.py#L89

[I didn't know `getopt` was exposed otherwise I'd use that , but there has to be a solution just using
argparse?]

Samuel Marks
Charity  | consultancy
 | open-source  |
LinkedIn 


Samuel Marks
Charity | consultancy | open-source | LinkedIn


On Fri, Oct 16, 2020 at 3:47 PM Dieter Maurer  wrote:
>
> Samuel Marks wrote at 2020-10-16 10:09 +1100:
> >Yes it’s my module, and I’ve been using argparse
> >https://github.com/SamuelMarks/ml-params
> >
> >No library I’ve found provides a solution to CLI argument parsing for my
> >use-case.
>
> Do you know that with `argparse` you can specify how many arguments an option
> expects? Thus, it should be quite easily possible to
> have --opt   ...
> Do you know that you can define new `Action`s for `argparse`?
> This way, you could properly process `--opt ,, ...`.
-- 
https://mail.python.org/mailman/listinfo/python-list