[issue9334] argparse does not accept options taking arguments beginning with dash (regression from optparse)
Tom Karzes added the comment: If it's going to be closed, it should at least be acknowledged that it *is* a fundamental design flaw, stemming from the misguided goal of trying (and necessarily failing) to allow options to be freely intermixed with positional arguments, which of course can't be done without dropping support for unrestricted string arguments. This means that argparse does not, and apparently never will, support string argument values that begin with hyphens. Yes, I know that the equal sign can be used to handle *some* cases, but who wants to use equal signs to specify command-line options? And there is no direct workaround for options that specify an nargs value of 2 or more. Also, fixing this is *not* hard at all. All that needs to be done is to add a keyword argument to the parser that tells it not to try to look ahead to find options, but to instead scan for them sequentially. Just like any properly designed option parser does. It's *easier* than trying to look ahead. I have my own quick-and-dirty hack that approximates this, but it's ugly and disgusting, it only handles some cases, and I hate using it. All the hack does is replace -- with a different prefix that I'm willing to avoid, then uses a different option prefix character so that the remaining strings starting with a single - are not seen as options. This handles a single leading - in an option value. It's not a general solution at all, but it's just barely good enough to solve my problem cases. Yes, argparse has been successful, but the reason for that success is that it wasn't made for proficient users. Rather, it was designed to cater to people who aren't very capable, and are perfectly happy to live with restricted string values if it means they can shuffle their command-line arguments willy nilly and still have it recognize them, rather than stopping when it should and giving the appropriate error. It trades precision for ease of use. This creates a vacuum for highly capable users who don't want to give up precise argument processing for the sake of a feature that's of no use to them in the first place. Don't get me wrong. There are a lot of nice features that argparse added which I find very useful. The problem is it also sacrifices core functionality, making its predecessor, optparse, preferable for some applications. In those cases, there is no transition path from optparse to argparse, since argparse does not handle all of the cases that optparse does. And since argparse does not subsume the functionality of optparse, I find the decision to deprecate optparse highly questionable. With optparse deprecated, Python no longer supports POSIX-compliant command-line option processing. How is that a sound decision? -- ___ Python tracker <https://bugs.python.org/issue9334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9334] argparse does not accept options taking arguments beginning with dash (regression from optparse)
Tom Karzes added the comment: Is there *still* no fix for this? I keep running into this bug. People sometimes say "oh, it's no problem, just use = to associate the option value with the option name". That is so sad. It's basically saying "it can't be made to work the way it should, so instead use = to introduce your option values." I should *never* have to use = to introduce an option value. And besides, using = doesn't even handle all cases. For example, suppose I have an option that takes two string arguments, i.e. type=str and nargs=2. Now I want to specify "-x" and "-y" as the two string arguments, like this: --opt -x -y As far as I can tell, argparse simply cannot handle this, and there's no workaround. Using = doesn't solve this case. One more time: All I want to do is disable the undesirable option look-ahead. It is utterly and completely useless to me. I want sequential, unambiguous option parsing. You know, the way the entire rest of the world does it. All that's needed is something that tells argparse to disable its look-ahead heuristic and to simply do what it's told. Scan left-to-right. If the next string is a recognized option name, then treat it as an option and take its arguments from the strings that follow, regardless of what they look like. Rinse and repeat. That is how correct option parsing is done. All this look-ahead heuristic does is cater to confused beginners, at the cost of breaking it for experienced users who know exactly what they want and are frustrated that argparse won't let them specify it. By the way, is there any supported, competing alternative to argparse? It seems like argparse is never going to support option values that begin with hyphens, so at this point I'm looking for an alternative that I don't have to fight every time I want to allow option values that begin with hyphens. Maybe it's time to create a new option parsing package that supports the most useful argparse features, but doesn't mistake option values for option names. You know, something more like optparse, but with some added features. It just needs to support strict left-to-right option parsing. At this point, I'm thinking it may be time to bite the bullet and write my own option parsing package. One that actually works, and can't be deprecated. But it seems like such a waste of time. It's hard to fathom why Python no longer provides a working option parser. -- ___ Python tracker <https://bugs.python.org/issue9334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41912] Long generator chain causes segmentation fault
New submission from Tom Karzes : If I create a sufficiently long chain of generators, I encounter a segmentation fault. For example, the following works as expected: % ./gen_bug3.py 1 1 % But for sufficiently larger chain lengths, it seg faults: % ./gen_bug3.py 2 Segmentation fault (core dumped) % and: % ./gen_bug3.py 10 Segmentation fault (core dumped) % The exact point where it seg faults seems to vary slightly between different invocations of Python, but the range is very narrow for a given Python installation. I believe the difference is due to slight variations in used memory upon startup. I can't see any reason why this should happen, and in any case, if there is some limit that I'm exceeding, it should raise an exception rather than core dump. I'm using: 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0] on a 64-bit Ubuntu Linux system. Additional info: A friend of mine is running 3.7.9 on a Windows system. In his case, the symptom is that the program produces no output for a sufficiently long generator chain (presumably it's silently crashing). Additionally, he encounters the problem with much shorter generator chains than I do. I suspect it's the same underlying problem. -- components: Interpreter Core files: gen_bug3.py messages: 377826 nosy: karzes priority: normal severity: normal status: open title: Long generator chain causes segmentation fault type: crash versions: Python 3.6 Added file: https://bugs.python.org/file49486/gen_bug3.py ___ Python tracker <https://bugs.python.org/issue41912> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41912] Long generator chain causes segmentation fault
Tom Karzes added the comment: That is a good point, except I don't believe the value needed to expose this bug is a "too-high limit" (as the documentation calls it). I set it to 100100 for convenience, but in practice even a value of 17000 is more than enough to expose the bug on my system (it occurs at around 16500). For my friend using Windows, a value as low as 4000 suffices, which I don't think anyone would argue is unreasonably high. The default value of 1000 is extremely low, and is intended to catch recursion bugs in programs that are not expected to recurse very deeply. For a program that genuinely needs recursion, I don't think a value of 2, or even 10, is unreasonable given today's typical memory sizes (and when I run my failing case, the memory usage is so low as to be inconsequential). By my interpretation, these limits should be well within the range that Python can handle. It seems likely to me that in this case, the problem isn't due to any kind of system limit, but is rather the result of a logical error in the implementation which is somehow exposed by this test. Hopefully a developer will take advantage of this test case to fix what I believe is a serious bug. -- ___ Python tracker <https://bugs.python.org/issue41912> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41912] Long generator chain causes segmentation fault
Tom Karzes added the comment: I tested this some more, and one thing became clear that I hadn't realized before: This bug has nothing to do specifically with generators (as I had thought), but is in fact due purely to the recursion limit. I created a recursive test program that doesn't use generators at all, and ran into the exact same problem (at only a slightly higher recursion level). I am attaching that test case as recurse_bug4.py. Here's a sample failing invocation: % ./recurse_bug4.py 2 Segmentation fault (core dumped) % On my system, the cutoff point for this one seems to be around 17600, roughly 1100 higher than for the generator example. What surprises me about this is the Python implementation uses recursion to implement recursion in Python apps. I would have thought that it would use heap memory for something like this, and as a result only require a fixed amount of stack. That would clearly have major advantages, since it would decouple recursion limits in Python code from stack limits on the platform. On the other hand, now that I understand that this is in fact the result of a system limit, I was *very* easily able to work around the problem! I simply disabled the stack limit. From csh: % unlimit stacksize % Then: % ./gen_bug3.py 10 10 % and: % ./recurse_bug4.py 10 10 % So you were right, this was due solely to the default stack limit on my system, and not a bug in the Python implementation. And it was also very easy to remove that limit. Hopefully something similar can be done on Windows (which I know very little about). Thank you for your help! -- Added file: https://bugs.python.org/file49487/recurse_bug4.py ___ Python tracker <https://bugs.python.org/issue41912> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41912] Long generator chain causes segmentation fault
Tom Karzes added the comment: Thanks Tim and Terry. Stackless Python sounds interesting. It's nice to know that others had the same idea I did, although I tend to shy away from exotic variants since they tend to be less well-supported. Any chance that CPython will go stackless at some point in the future? By the way, I saw that I can increase the process stack limit from within a Python app: resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY)) I tried it, and it works (on my Linux system), but of course is unavailable on Windows systems. -- ___ Python tracker <https://bugs.python.org/issue41912> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9334] argparse does not accept options taking arguments beginning with dash (regression from optparse)
Tom Karzes added the comment: I'm dismayed to see that this bug was reported in 2010, yet as of January 2018 has not yet been fixed. This option parsing behavior is contrary to Unix option passing conventions. I certainly don't mind enhancements, but the last thing that should *ever* be introduced into an option parser is ambiguity. It's dangerous. I'm an old-school Unix guy. I know exactly what options I want to pass to my program, and I know exactly how they should be passed. Options first, with option arguments immediately following the option name, followed by positional arguments (with no more options being recognized after that point). If the first positional argument begins with a hyphen, "--" can be used to end option parsing. That's all I want. Simple. Precise. Unambiguous. And it doesn't place any constraints on what an option value can look like. Yes, I know I can get around the problem by using "=" to join option values to the preceding option names, but that shouldn't be necessary. In my opinion, the entire approach of attempting to distinguish option names from arguments out of context is fatally flawed. It cannot be done. Period. Furthermore, it's utterly unnecessary. All I want is a parameter I can pass to argparse to tell it to accept the entire command line at face value, without any second-guessing. If I specify an option that takes an integer argument, then the very next command line argument should be an integer. If it isn't, I want an immediate error. More to the point, if I specify an option that takes a string argument, then the very next command line argument is that string, period. It doesn't matter if it begins with a hyphen or any other character for that matter. It's a string. It can contain *any* character. And yet, argparse doesn't support this basic, fundamental functionality without the user inserting an "=" to join the arguments. Why?? Here's an analogy: Your favorite car company has released a new car. It adds lots of great features, like a sun roof and a fancy navigation system. But oops, the brakes no longer work if you're turning right when you apply them. But not to worry! You can always just throw it into park to stop the car. It's worth it for the new features, right? Wrong. It seems to me like there are three possible solutions to this: (1) Add a "traditional" mode to argparse that completely bypasses any attempt to classify command line arguments as "options" vs. "arguments", (2) Un-deprecate optparse, and resume development on it, adding support for some of the argparse features but without breaking standard Unix option parsing, or (3) Create yet another new option parsing package for Python, one which supports traditional Unix option parsing and doesn't introduce gratuitous ambiguities and restrictions on what strings can contain. My specific case: I have an option whose argument is a comma-separated list of signed integers, as a single string. This is the format another, non-Python application requires, and it needs to be compatible. But the following fails: --myopt -1,2 Apparently it thinks "-1,2" is an option name. No, it's a string option value, and should not be interpreted in any other way. I want it passed as the option value. I would want the same no matter *what* characters it contained. And I don't want to have to glue the two arguments together with "=". -- nosy: +karzes ___ Python tracker <https://bugs.python.org/issue9334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9334] argparse does not accept options taking arguments beginning with dash (regression from optparse)
Tom Karzes added the comment: Here's my situation: I originally used optparse, although some of the guys I worked with at the time were starting to use argparse. At first I thought, I'm sticking with optparse, it's more standard than argparse and probably better supported. But at some point optparse became documented as deprecated, with argparse being hailed as its replacement. At that point my preference switched, again mostly because I wanted the most reliable, best supported option parsing package. And yes, I have come to appreciate some of the features of argparse, but to me that takes a back seat to correct functionality. The documentation for argparse promotes the idea that it completely subsumes optparse, and that applications that use optparse can easily be converted to use argparse. And for the most part that's true, except that optparse identifies options correctly and argparse does not. That really, really needs to be documented. What I want is a supported, standard option parsing library that knows how to extract option values correctly. I used to have that with optparse, but now I feel like the rug's been pulled out from under me. optparse is supposedly deprecated, and argparse doesn't work. So I either use a package that could be removed from Python distributions at any time, or I use a package that has dangerous bugs. I don't find either alternative very attractive. As I said, un-deprecating optparse would be sufficient, especially if people started adding some of the argparse functionality to it. Creating a new package would work too. But from what I've seen, it sounds like argparse is beyond hope of repair and will never work properly. I.e., it's a dead-end development path. So why isn't *argparse* deprecated? I've always maintained that core functionality is of primary importance. Shiny bells and whistles, no matter how useful, are of secondary importance. In my opinion, the wrong package was deprecated. -- ___ Python tracker <https://bugs.python.org/issue9334> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
