[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin added the comment: Just following up on this. Now that 3.2 is out, has the patch been committed? -- ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin added the comment: Don't mean to nag. Just checking to see if anyone has taken it upon themselves to commit this yet since it's been a couple more months :P -- ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies
New submission from Justin : This is the same issue as was reported here: http://bugs.python.org/issue1954. It is still a problem in Python 3.1. I'm writing a server that will be receiving a massive number of requests and I'd like to eliminate the zombie problem. Once I figured out what was going on, I tried adding a call to collect_children() in the serve_forever() loop. It worked very well. I've included a patch of what I did, however, I obviously can't leave this change in my socketserver.py because we will be deploying this on a lot of servers. Is there any reason not to collect_children() in the serve_forever() loop? It seems like the place to do it to me. The patch will only collect children if there are any, so it doesn't have to call it every time through the loop. -- components: Library (Lib) files: collect_children.patch keywords: patch messages: 127825 nosy: jwark priority: normal severity: normal status: open title: socketserver.ForkingMixIn leaves zombies type: behavior versions: Python 3.1 Added file: http://bugs.python.org/file20665/collect_children.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies
Justin added the comment: I guess I didn't really explain the issue much. The problem is that if the server receives say, 10 requests at once, and then forks a process for each, after those processes finish they will sit as zombies until process_request() is called again, which calls collect_children(). So that patch simply moves the call to collect_children() into the serve_forever() loop so that it doesn't leave zombies sitting around. -- ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies
Justin added the comment: Good point. I was just writing up something quick that works. Here's another patch, is this acceptable? -- Added file: http://bugs.python.org/file20690/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin added the comment: I noticed that ForkingMixIn also was overriding handle_timeout() trying to cleanup zombies after 300 seconds of inactivity, which is useless on a busy server. I'm replacing the patch with one that also removes handle_timeout(). -- Added file: http://bugs.python.org/file20691/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Changes by Justin : Removed file: http://bugs.python.org/file20690/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Changes by Justin : Removed file: http://bugs.python.org/file20691/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Changes by Justin : Added file: http://bugs.python.org/file20693/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin added the comment: I hope I did that last patch right. I did a 'diff -u' instead of a 'diff -c'. If you need something different, let me know. -- ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin added the comment: Sorry I keep plaguing this with comments and files, but I got to thinking, anyone should be able to override _loop_actions() and implement what they need in the loop. While it's probably a bad idea in most cases, there may be legitimate needs and it's always good to allow the flexibility. So, I'm adding one more patch that changes the name to loop_actions() and adds it to the list of methods that can be overridden in the BaseServer docstring. If you like it, keep it, if not, just use the last one. -- Added file: http://bugs.python.org/file20695/loop_actions.patch ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42640] tkinter throws exception when key is pressed
New submission from Justin : On macOs 10.14.6 laptop with a qwerty layout, when I switch my keyboard layout to `French - PC` and use a tkinter keyboard listener and press the '[' button which should be the '^' button on the azerty keyboard, this Error is thrown: ``` 2020-12-14 10:13:26.533 Python[14880:1492926] *** Terminating app due to uncaught exception 'NSRangeException', reason: '-[__NSCFConstantString characterAtIndex:]: Range or index out of bounds' *** First throw call stack: ( 0 CoreFoundation 0x7fff41ce49ad __exceptionPreprocess + 256 1 libobjc.A.dylib 0x7fff6c3dfa17 objc_exception_throw + 48 2 CoreFoundation 0x7fff41ce47df +[NSException raise:format:] + 201 3 CoreFoundation 0x7fff41c5a159 -[__NSCFString characterAtIndex:] + 102 4 Tk 0x7fff4da8a806 TkpInitKeymapInfo + 719 5 Tk 0x7fff4da905e9 Tk_MacOSXSetupTkNotifier + 793 6 Tcl 0x7fff4d98c48e Tcl_DoOneEvent + 301 7 _tkinter.cpython-38-darwin.so 0x0001090802de _tkinter_tkapp_mainloop + 342 8 Python 0x000108ac1b0a method_vectorcall_FASTCALL + 250 9 Python 0x000108b5a299 call_function + 346 10 Python 0x000108b51457 _PyEval_EvalFrameDefault + 3895 11 Python 0x000108b5ae5d _PyEval_EvalCodeWithName + 2107 12 Python 0x000108abad39 _PyFunction_Vectorcall + 217 13 Python 0x000108abcc7d method_vectorcall + 135 14 Python 0x000108b5a299 call_function + 346 15 Python 0x000108b51477 _PyEval_EvalFrameDefault + 3927 16 Python 0x000108b5ae5d _PyEval_EvalCodeWithName + 2107 17 Python 0x000108b5047d PyEval_EvalCode + 51 18 Python 0x000108b89025 run_eval_code_obj + 102 19 Python 0x000108b88473 run_mod + 82 20 Python 0x000108b87345 PyRun_FileExFlags + 160 21 Python 0x000108b86a29 PyRun_SimpleFileExFlags + 271 22 Python 0x000108b9e449 Py_RunMain + 1870 23 Python 0x000108b9e790 pymain_main + 306 24 Python 0x000108b9e7de Py_BytesMain + 42 25 libdyld.dylib 0x7fff6dbae3d5 start + 1 26 ??? 0x0002 0x0 + 2 ) libc++abi.dylib: terminating with uncaught exception of type NSException Abort trap: 6 ``` One can verify this by running this program on a macOs laptop with a qwerty keyboard, switching the layout to French - PC and pressing the '[' key. ``` import pygame pygame.init() pygame.display.set_mode((100, 100)) while True: for event in pygame.event.get(): if event.type == pygame.QUIT: pygame.quit(); #sys.exit() if sys is imported if event.type == pygame.KEYDOWN: key_name = pygame.key.name(event.key) print(event, event.key.__class__, event.key, key_name) elif event.type == pygame.KEYUP: key_name = pygame.key.name(event.key) print(event, event.key.__class__, event.key, key_name) ``` ``` -- messages: 382998 nosy: spacether priority: normal severity: normal status: open title: tkinter throws exception when key is pressed ___ Python tracker <https://bugs.python.org/issue42640> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42640] tkinter throws exception when key is pressed
Justin added the comment:
Sorry my verification keyboard listener program is not the pygame one. It is
this one:
```
from tkinter import *
def keyup(e):
print('up', e.__dict__)
def keydown(e):
print('down', e.__dict__)
root = Tk()
frame = Frame(root, width=100, height=100)
frame.bind("", keydown)
frame.bind("", keyup)
frame.pack()
frame.focus_set()
root.mainloop()
```
--
___
Python tracker
<https://bugs.python.org/issue42640>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42640] tkinter throws exception when key is pressed
Justin added the comment: Thanks for responding so quickly. 1. My python version is: Python 3.8.6 (default, Oct 8 2020, 14:07:53) [Clang 11.0.0 (clang-1100.0.33.17)] on darwin 2. N/A 3. >>> from tkinter.test.support import * >>> get_tk_patchlevel() (8, 5, 9, 'final', 0) 4. No; the pygame sample does not crash with this key input Which version of TK 8.5.X fixes this, 8.5.18.0? That's what it looks like here: https://www.python.org/download/mac/tcltk/ -- ___ Python tracker <https://bugs.python.org/issue42640> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42640] tkinter throws exception when key is pressed
Justin added the comment: FYI, I just brew installed python and with: ``` Python 3.9.1 (default, Dec 17 2020, 03:56:09) [Clang 11.0.0 (clang-1100.0.33.17)] on darwin ``` This issue still happens -- ___ Python tracker <https://bugs.python.org/issue42640> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42640] tkinter throws exception when key is pressed
Justin added the comment: Thank you very much. I understand and just wanted to let you know. In brew I opened up this ticket(https://github.com/Homebrew/homebrew-core/issues/67327) with that team. -- ___ Python tracker <https://bugs.python.org/issue42640> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42695] tkinter keysym_num value is incorrect
New submission from Justin :
Hi there.
On my MacOS 10.14.16 laptop with a qwerty keyboard I was testing tkinter
keyboard listening for an azerty keyboard layout by switching the layout to
`French - PC`
When I press qwerty keys Shift + \ I expect to see 'μ' printed.
When looking at the tkinter events for that key press we see:
```
down {'keysym': 'Shift_L', 'keysym_num': 65505, 'keycode': 131072, 'char': ''}
down {'keysym': 'tslash', 'keysym_num': 956, 'keycode': 956, 'char': 'μ'}
up {'keysym': 'asterisk', 'keysym_num': 42, 'keycode': 2753468, 'char': 'μ'}
up {'keysym': 'Shift_L', 'keysym_num': 65505, 'keycode': 131072, 'char': ''}
```
So the char value is correct but the keysym_num is not the expected 181 for mu.
Comparing this to pressing Shift + / to generate the section symbol (§) we see:
```
down {'keysym': 'Shift_L', 'keysym_num': 65505, 'keycode': 131072, 'char': ''}
down {'keysym': 'section', 'keysym_num': 167, 'keycode': 167, 'char': '§'}
up {'keysym': 'section', 'keysym_num': 167, 'keycode': 2883751, 'char': '§'}
up {'keysym': 'Shift_L', 'keysym_num': 65505, 'keycode': 131072, 'char': ''}
```
Which produces the expected keysym_num of 167.
TLDR: the kysym_num value when writing the mu character is incorrect. It should
be 181 and logging shows values of 956 and 42. Can this be fixed?
Here is the keyboard listener program which can be used for verification:
```
from tkinter import *
params = ['keysym', 'keysym_num', 'keycode', 'char']
def keyup(e):
d = {p: getattr(e, p) for p in params}
print('up', d)
# print('up', e.__dict__)
def keydown(e):
d = {p: getattr(e, p) for p in params}
print('down', d)
# print('down', e.__dict__)
pass
root = Tk()
frame = Frame(root, width=100, height=100)
frame.bind("", keydown)
frame.bind("", keyup)
frame.pack()
frame.focus_set()
root.mainloop()
```
Note: my python version was installed from python.org and is:
```
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec 7 2020, 12:44:01)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
```
--
messages: 383458
nosy: spacether
priority: normal
severity: normal
status: open
title: tkinter keysym_num value is incorrect
___
Python tracker
<https://bugs.python.org/issue42695>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42695] tkinter keysym_num value is incorrect
Justin added the comment: TK bug ticket has been created at https://core.tcl-lang.org/tk/tktview/ffe6925b916caac02acae53f745e95dd1c557019 -- ___ Python tracker <https://bugs.python.org/issue42695> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35531] xml.etree.ElementTree Elment.find bug, fails to find tag
New submission from Justin :
When the following text it loaded in to an ElementTree Element, the find method
is unable to find one of the elements without a namespace assigned to it.
```
import xml.etree.ElementTree as ElementTree
xml_text = """
http://schemas.xmlsoap.org/soap/envelope/";>a:ActionNotSupportedThe message with Action \'\' cannot be processed at the
receiver, due to a ContractFilter mismatch at the EndpointDispatcher. This may
be because of either a contract mismatch (mismatched Actions between sender and
receiver) or a binding/security mismatch between the sender and the receiver.
Check that sender and receiver have the same contract and the same binding
(including security requirements, e.g. Message, Transport,
None).
"""
xml = ElementTree.fromstring(xml_text)
ele = xml.find('faultstring')
ele == None #True
```
--
components: XML
messages: 332106
nosy: spacether
priority: normal
severity: normal
status: open
title: xml.etree.ElementTree Elment.find bug, fails to find tag
type: enhancement
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue35531>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35531] xml.etree.ElementTree Elment.find bug, fails to find tag
Justin added the comment:
Issue was user error. I though that find did a full search of the tree when it
only searches children.
Solution is:
ele = xml.find('.//faultstring')
--
resolution: -> not a bug
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue35531>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29837] python3 pycopg2 import issue on solaris 10
New submission from justin: Hi, I have installed psycopg2 through pip3, but when I tried to import it, I got the following error. what could be the problem? help> psycopg2 problem in psycopg2 - ImportError: ld.so.1: python3.3: fatal: relocation error: file /opt/csw/lib/python3.3/site-packages/psycopg2/_psycopg.so: symbol timeradd: referenced symbol not found thanks justin -- components: Build messages: 289763 nosy: juwang priority: normal severity: normal status: open title: python3 pycopg2 import issue on solaris 10 type: compile error versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue29837> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7190] TCP/IP?
New submission from Justin : Mac OS X IBook G4 PowerPC when running target.py it works fine until i close the app without ending the process. (before 'q' is entered by the user). sometimes python will not allow the user to click in the window and have the ability to type after running the target window. sometimes the mouse clicks affect a part of IDLE up and left of where the actual click was. when i ignore this error (this is the only thing i have open in IDLE) i must fist close out of python and reopen target.py i have used this approach to test and find that there are no bugs in my program (at least it works above and beyond the specifications of my class assignment), but every once in while i will get an error when trying to use 'q' to end the program and python will state that win.getMouse() is not a viable term to use, because (win) is already closed yet when 'q' is used the if statement has the close command right underneath the get mouse command. *note* i have coded another program specifically from John Zelle's "Python Programming: An Introduction to Computer Science" p.160 programming exercise #1 and my get display text, get mouse, then close window is verbatim from that in target.py and it never showed an error in the #1's program IDLE can't bind TCP/IP port 8833, which is necessary to communicate with its Python execution server. Either no networking is installed on this computer or another process (another IDLE?) is using the port. Run IDLE with the -n command line switch to start without a subprocess and refer to Help/IDLE Help 'Running without a subprocess' for further details. -- assignee: ronaldoussoren components: Macintosh files: target.py messages: 94385 nosy: jlr2383, ronaldoussoren severity: normal status: open title: TCP/IP? type: crash versions: Python 2.6 Added file: http://bugs.python.org/file15187/target.py ___ Python tracker <http://bugs.python.org/issue7190> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23429] -5**4 returns -625 instead of 625
New submission from Justin: C:\Users\Justin>python Python 3.4.2 (v3.4.2:ab2c023a9432, Oct 6 2014, 22:15:05) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> print(-5**4) -625 >>> #note...this should be 625 and not a negative 625 -- messages: 235642 nosy: gilbe024 priority: normal severity: normal status: open title: -5**4 returns -625 instead of 625 type: behavior versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue23429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23429] -5**4 returns -625 instead of 625
Justin added the comment: I think there is a misunderstanding of precedence here. If we had an operation were -1 * 5**4, I could fully understand the statement of precedence. However, in the absence of the "-1 *", the -5 becomes a single atom. For example: >>> myVal = -5 >>> myVal**4 625 >>> # not -625 because it is treated as a single atom. The statement -5**4 - 20, should yield the same result and yet, it does not. >>> -5**4 - 20 -645 >>> 5**4 - 20 605 >>> Let's take the reverse side of this take 4**-2, if precedence is to hold the same way, why would this not be 16 instead of the correct answer of 0.0625. In this instance, the unary sign all of a sudden jumped in precedence and became a single atom. >>> 4**-2 0.0625 -- ___ Python tracker <http://bugs.python.org/issue23429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23429] -5**4 returns -625 instead of 625
Justin added the comment: Thank you for the link. I can understand the language definition and I can accept it at face value. The confusion I have is that it does not appear to be consistent. My example of -5**4 vs 4**-2 would both have literals with a unary sign. One works as expected, whereas the other one does not. Granted that I am new to Python, although well experience in development and programming; I will conceded to those more experienced in Python development that it is as it should be. Regards, -- ___ Python tracker <http://bugs.python.org/issue23429> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1220] popen3 website documentation inconsistency
New submission from Justin Bronn: The Python website says that the following tuple is returned from popen3: "Returns the file objects (child_stdout, child_stdin, child_stderr)." See http://docs.python.org/lib/module-popen2.html. However, the docstring of popen3 gets the order right (using Python 2.5.1): "The file objects (child_stdin, child_stdout, child_stderr) are returned." See `help(os.popen3)`. -- components: Documentation messages: 56198 nosy: jbronn severity: normal status: open title: popen3 website documentation inconsistency versions: Python 2.5 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1220> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13031] [PATCH] small speed-up for tarfile.py when unzipping tarballs
New submission from Justin Peel : Attached small diff speeds up extracting a gzipped tarball on my machine using python 3.2 by 3-5%. It will probably be a larger percentage on machines that have faster hard drives (mine is 5400rpm). Basically, the changes speed up calculating the checksums by only doing one slice rather than four and call struct.unpack twice rather than four times. We are able to use less unpack calls because 'x' means to skip a byte. -- components: Library (Lib) files: cpython_tarfile.diff keywords: patch messages: 144436 nosy: jpeel priority: normal severity: normal status: open title: [PATCH] small speed-up for tarfile.py when unzipping tarballs versions: Python 2.7, Python 3.2 Added file: http://bugs.python.org/file23229/cpython_tarfile.diff ___ Python tracker <http://bugs.python.org/issue13031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13031] [PATCH] small speed-up for tarfile.py when unzipping tarballs
Changes by Justin Peel : -- type: -> performance ___ Python tracker <http://bugs.python.org/issue13031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13031] small speed-up for tarfile.py when unzipping tarballs
Justin Peel added the comment: poq, You're quite right. I've added that change too. By the way, four unnecessary extra tuples are no longer being created for each call to this function too because of these changes. -- Added file: http://bugs.python.org/file23232/cpython_tarfile2.diff ___ Python tracker <http://bugs.python.org/issue13031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils.filelist / packaging.manifest test coverage
Justin Love added the comment: Sorry, it's been sitting at the bottom of my starred list for months, the prospect of reloading context always making it a little less attractive than something else. I turned out I had to update to even have 'packaging' Error was caused by an actual change in behavior - I was testing the argument-check by providing an action with no argument. The new code assumes a single 'word' is a filename, and translates it to include, eg. 'include include' which then passes the parse. I modified the Manifest to test against the list of actions, since this makes it harder for users to make a mistake. Of course we could just drop the test, or only check multi-argument actions. -- Added file: http://bugs.python.org/file23235/packaging-manifest_v2.diff ___ Python tracker <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils.filelist / packaging.manifest test coverage
Justin Love added the comment: Incorporated some formatting suggestions. Removed _actions. Just as well, as I accidentally left in the 'blarg' bogus action ;^) Added a test for implicit-include. Whether 'this cannot happen' can happen depends on the output of _parse_template_line. If you wanted to remove the catch-all and still have test backup, you'd probably have to restort to some form of fuzz-testing _process_template_line. As the code stands now it could be removed. I left it alone as insurance against future changes. -- Added file: http://bugs.python.org/file23244/packaging-manifest_v3.diff ___ Python tracker <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils.filelist / packaging.manifest test coverage
Justin Love added the comment: re: test_glob_to_re Could not reproduce (OS X 10.6) I built branch 2.7, and also tried the expressions from glob_to_re on my system 2.7 and 3.2, all returned the (?ms) version. -- ___ Python tracker <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils.filelist / packaging.manifest test coverage
Justin Love added the comment: Still can't reproduce (though I got one failure and three other errors) $ patch -p 1 <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11109] socketserver.ForkingMixIn leaves zombies, also fails to reap all zombies in one pass
Justin Warkentin added the comment: Sorry, I haven't had a chance to look at this in a couple days. I've been very busy with work. I'm not sure exactly how to write the test for this, so I don't know that I'd be much help there. One last thing, I was just looking through the commits and I noticed in the NEWS update (http://hg.python.org/cpython/rev/3e3cd0ed82bb/) you have my name as Justin Wark. The last name is actually Warkentin, I just didn't have anything showing that. That's my fault, sorry. I just updated my profile to show my full name. -- ___ Python tracker <http://bugs.python.org/issue11109> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12537] mailbox's _become_message is very fragile
Changes by Justin Wehnes : -- nosy: +jwehnes ___ Python tracker <http://bugs.python.org/issue12537> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10415] readline.insert_text documentation incomplete
New submission from Justin Lebar : The readline documentation currently says: > readline.insert_text(string) > Insert text into the command line. But as far as I can tell, readline.insert_text() does something only when called from startup_hook or pre_input_hook. Here's an example of someone using the module in a way that works: http://swapoff.org/svn/cly/tags/0.7/cly/interactive.py -- assignee: d...@python components: Documentation messages: 121178 nosy: Justin.Lebar, d...@python priority: normal severity: normal status: open title: readline.insert_text documentation incomplete versions: Python 3.1 ___ Python tracker <http://bugs.python.org/issue10415> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10415] readline.insert_text documentation incomplete
Justin Lebar added the comment: Actually, maybe startup_hook doesn't do what it sounds like it does and insert_text() only works from startup_hook. If this is the case, then the documentation for startup_hook could also be improved: > The startup_hook function is called with no arguments just before > readline prints the first prompt. -- ___ Python tracker <http://bugs.python.org/issue10415> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils/filelist test coverage
New submission from Justin Love : Increase test coverage of distutils/filelist.py from 49% to 100%. One line was marked as excluded because it was a "this cannot happen" error, and I agreed. -- components: Tests files: increase_distutils_filelist_test_coverage.patch keywords: patch messages: 132869 nosy: jlove priority: normal severity: normal status: open title: Increase distutils/filelist test coverage type: behavior Added file: http://bugs.python.org/file21518/increase_distutils_filelist_test_coverage.patch ___ Python tracker <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11751] Increase distutils.filelist test coverage
Justin Love added the comment: Removed NO COVER Combined test_translate_pattern Added on to some test names to make them more descriptive -- Added file: http://bugs.python.org/file21562/increase_distutils_filelist_test_coverage_v2.patch ___ Python tracker <http://bugs.python.org/issue11751> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7995] On Mac / BSD sockets returned by accept inherit the parent's FD flags
Justin Cappos added the comment: > Apparently, the designers of BSD thought differently. Remember that > it is them who defined the socket API in the first place, so they > have the right that their design decisions are considered. I think there is a bit of confusion here. The 'bug' isn't with different socket semantics on different OSes. The bug is that the programmer who wrote the wrapper for sockets on Python assumed the OS semantics weren't the BSD style. Here is the issue (put plainly): Python sockets support a notion of timeout (note this notion is not reflected in the OS socket API). The python socket implementation of timeouts uses the underlying OS / socket API to provide this by setting the socket to nonblocking and setting a timeout value in a Python object that holds socket info. This implementation assumes that the OS sets any socket it receives via accept to nonblocking. (this is a false assumption on BSD) The end result is that the OS has a nonblocking socket and the Python object thinks it is blocking. This is why the socket object in Python has timeout=None yet calling fcntl shows the socket is nonblocking. Calling code paths that handle timeouts and expect the socket to block causes bugs like I described in my code. This behavior is clearly wrong under any interpretation! You can debate whether the right patch is to use what I proposed or instead change new Python sockets to inherit the timeout / blocking setting on BSD. However, what is implemented now is clearly broken. -- ___ Python tracker <http://bugs.python.org/issue7995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7995] On Mac / BSD sockets returned by accept inherit the parent's FD flags
Justin Cappos added the comment: >> This implementation assumes that the OS sets any socket it receives >> via accept to nonblocking. (this is a false assumption on BSD) > > Not true. It doesn't assume that (it doesn't assume the reverse, > either). The Python implementation sets timeout=None (which implies that the underlying socket is blocking). >> The end result is that the OS has a nonblocking socket and the Python >> object thinks it is blocking. This is why the socket object in >> Python has timeout=None yet calling fcntl shows the socket is >> nonblocking. > > That conclusion is flawed. Python has not associated a timeout with > the socket. It makes no claims as to whether the socket is blocking > or not. So you have created a non-blocking socket without timeout. The problem is that it has. It has created a new Python socket object with a specific value for timeout (None), but the underlying socket is nonblocking. The docs state that timeout = None makes the socket blocking. -- ___ Python tracker <http://bugs.python.org/issue7995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7995] On Mac / BSD sockets returned by accept inherit the parent's FD flags
Justin Cappos added the comment: > > Well, I don't think setting a timeout on a listening socket and then > > expecting the socket received through accept() to be non-blocking (but > > only on BSD) is a legitimate application. > > > Right. But setting the server socket to nonblocking, and then > expecting the connection socket to also be nonblocking might be. Okay sure. This is fine. That is why I suggested that if you don't like my patch, one might instead change new Python sockets to inherit the timeout / blocking setting on BSD. However, I hope we can all agree that having the Python socket object in a different blocking / non-blocking state than the OS socket descriptor is wrong. This is what needs to be fixed. -- ___ Python tracker <http://bugs.python.org/issue7995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7995] On Mac / BSD sockets returned by accept inherit the parent's FD flags
Justin Cappos added the comment: >> The Python implementation sets timeout=None (which implies that the >> underlying socket is blocking). > >No, it doesn't. A socket may be non-blocking without having a timeout; > that's the socket API (on all systems, not just BSD). Sure and this happens when the timeout is 0, but None has a different meaning than 0. >> The problem is that it has. It has created a new Python socket >> object with a specific value for timeout (None), but the underlying >> socket is nonblocking. >> >> The docs state that timeout = None makes the socket blocking. > > What specific wording are you looking at that makes you believe so? Here is the last part of the description of settimeout: s.settimeout(None) is equivalent to s.setblocking(1) if you look at setblocking: Set blocking or non-blocking mode of the socket: if flag is 0, the socket is set to non-blocking, else to blocking mode. This seems to imply that timeout = None -> blocking. -- ___ Python tracker <http://bugs.python.org/issue7995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10667] collections.Counter object in C
New submission from Justin Peel :
I put the Counter's update and __missing__ methods into C code. The rest of the
existing methods remain the same. The new Counter is at least 2x-10x (or more)
faster for updating depending on the input. I also added a new method,
update_fromsubs, which is documented below the timings for updating a Counter.
Here are a few examples to demonstrate the speed of the new update function:
Counting list(range(100))*100 -
Old Counter: 18.6 msec
New Counter: 1.43
13x speed-up
Counting range(1) -
Old Counter: 21.5 msec
New Counter: 3.84 msec
5.6x speed-up
Counter generator i % 100 for i in range(1) -
Old Counter: 36.7 msec
New Counter: 17.5 msec
2.1x speed-up
and the __missing__ method being in C makes getting missing keys about twice as
fast.
Also, I added a new method, update_fromsubs, which counts subarrays/substrings
of a sequence. This method only works with immutable sequences like tuples,
bytes, and unicode objects. The method is of the form
update_fromsubs(seq, frame[, step[, lo[, hi]]])
where frame is the size of the subarrays, step is how far to move forward after
each subarray, and lo and hi are the respective starting and ending indices to
count subarrays within the sequence.
For instance,
c = Counter()
c.update_fromsubs('abcd', 2)
yields
Counter({'cd': 1, 'ab': 1, 'bc': 1})
and
d = Counter()
d.update_fromsubs('abcd', 2, 2)
yields
Counter({'ab': 1, 'cd: 1})
These sorts of operations could be done with generators in a manner like
Counter(s[i:i+2] for i in range(0, len(s)-1))
but it can be about twice as fast by using update_fromsubs. Here's an example
Counting subarrays of length 2 of "ab"*1:
update_fromsubs:30.8 ms
New Counter w/ generator: 54.3 ms
Old Counter w/ generator: 98.8 ms
This method would probably be most useful in processing strings, but there may
be other uses. If it is accepted, please feel to change the method name and
parameters. I especially wasn't sure what to call this method
(update_fromsubsequences seemed rather long).
--
components: Extension Modules, Library (Lib)
files: counterpatch.diff
keywords: patch
messages: 123706
nosy: jpeel, rhettinger
priority: normal
severity: normal
status: open
title: collections.Counter object in C
type: performance
versions: Python 3.3
Added file: http://bugs.python.org/file19992/counterpatch.diff
___
Python tracker
<http://bugs.python.org/issue10667>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10667] collections.Counter object in C
Justin Peel added the comment:
I've done as Antoine asked and made a pure Python PyCounter class and a
C-enhanced Counter class that both use the mixin CounterBase class. I also
added to the tests so that both PyCounter and Counter are tested. I left the
update_fromsubs() method in Counter for now, but haven't made a Python
equivalent function for PyCounter in case the method is rejected.
Also, I realized that my build was still in debug mode so those benchmark
weren't quite accurate (Oops). The Counter class is only 4-6x faster for most
cases. That's still good, but not quite as good as I'd previously reported.
Lastly, I thought of one more method that could be quite useful. Say that you
have a list like [('apples',4),('oranges',5),('apples',8)] where the second
element in each tuple indicates how many of them there are. The resultant
Counter should have {'apples': 12, 'oranges':5}. We don't really have a good
way to count this, but we could add a small method like the following:
def fromitems(self, items):
for key, value in items:
self[key] += value
and I could easily make a C function for it. I think that this simple method
could be very useful to some people. What do you all think of this addition?
--
Added file: http://bugs.python.org/file20004/counterpatch2.diff
___
Python tracker
<http://bugs.python.org/issue10667>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10667] collections.Counter object in C
Justin Peel added the comment: Okay, I was done submitting. I thought that Antoine was asking for a version that kept a pure Python version so I did that. -- ___ Python tracker <http://bugs.python.org/issue10667> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8637] Add PYDOCPAGER envvar to specify pager for pydoc
Justin Bronder added the comment: On Thu, Feb 3, 2011 at 7:39 PM, Éric Araujo wrote: .. > Justin: what exactly is a pager that does not support man pages? vimpager is the one I'm using. For man pages there's vimmanpager. -- ___ Python tracker <http://bugs.python.org/issue8637> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8637] Add PYDOCPAGER envvar to specify pager for pydoc
Justin Bronder added the comment: > Éric Araujo added the comment: > > Excuse me if I was unclear: I wasn’t asking for program names but for a > definition of “don’t support” or description of problematic behavior. In headers with vimpager ^H is not rendered(?) correctly. For instance, 'pydoc os' shows: --- Help on module os: N^HNA^HAM^HME^HE F^HFI^HIL^HLE^HE /usr/lib/python2.6/os.py --- And so on. -- ___ Python tracker <http://bugs.python.org/issue8637> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3144] popen / popen[234] inconsistent fd behavior
New submission from Justin Cappos <[EMAIL PROTECTED]>:
The behavior of popen vs popen[2-4] differs with respect to open file
descriptors (at least on the Linux implementation of popen). popen
does not close file descriptors, thus processes retain open file
descriptors from their parent. This is likely not desirable for
security and stability reasons.
If this isn't fixed, at a minimum it would be a good thing to document.
Here is an example that demonstrates the issue:
<<< start of open_and_popen.py>>>
# This will not be printed if popen closes file descriptors
import os
myfd = os.open("open_and_popen.py",os.O_RDONLY)
readfo = os.popen("python print_from_fd.py "+str(myfd),"r")
print "os.popen results in:"
print readfo.read()
# it will print the first line of the file here
readfo.close()
(junkinfo, readfo) = os.popen2("python print_from_fd.py "+str(myfd),"r")
junkinfo.close()
print "os.popen2 results in:"
print readfo.read()
# the child got an error, so this is just the error text
readfo.close()
os.close(myfd)
<<< end of open_and_popen.py>>>
<<< start of print_from_fd.py>>>
import os
import sys
print os.read(int(sys.argv[1]),60)
<<< end of print_from_fd.py>>>
--
components: Library (Lib)
messages: 68416
nosy: justincappos
severity: normal
status: open
title: popen / popen[234] inconsistent fd behavior
type: security
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue3144>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3335] subprocess lib - opening same command fails
New submission from Justin Harper <[EMAIL PROTECTED]>:
When I make the same shell call a second time, a weird error occurs.
It appears that the system is caching the call and then returning the
same object the second time, which causes a problem because the stream
is at EOF and there is no way to seek on this sort of file object.
Code:
fout = subprocess.Popen("owplaces -silent -multi", shell=True,
stdout=subprocess.PIPE).stdout
self.output = fout.read()
if self.output != []:
for line in self.output:
print line
fout.close()
Error:
Traceback (most recent call last):
File "./saveSettings.py", line 30, in next_page
func()
File "./saveSettings.py", line 62, in save_startup
fout = subprocess.Popen("owplaces -silent -multi", shell=True,
stdout=subprocess.PIPE).stdout
File "/opt/app/g++lib6/python-2.4/lib/python2.4/warnings.py", line
61, in warn
warn_explicit(message, category, filename, lineno, module, registry)
File "/opt/app/g++lib6/python-2.4/lib/python2.4/warnings.py", line
82, in warn_explicit
for item in filters:
TypeError: an integer is required
The first time the code works time. The second (and subsequent times)
the cryptic error msg is displayed.
Python 2.4.5 running on a Solaris machine.
--
components: Library (Lib)
messages: 69514
nosy: gtg944q
severity: normal
status: open
title: subprocess lib - opening same command fails
versions: Python 2.4
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue3335>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3335] subprocess lib - opening same command fails
Justin Harper <[EMAIL PROTECTED]> added the comment: Correction: line two should actually be: self.output = fout.readlines() The bug still exists with this change. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3335> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3335] subprocess lib - opening same command fails
Justin Harper <[EMAIL PROTECTED]> added the comment: Further research reveals that the problem appears to be with the gtk.Assistant class in PyGTK 2.10, not the subprocess lib. Report can be closed. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3335> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7746] Nit: Extra comma in itertools doc
New submission from Justin Lebar : At http://docs.python.org/3.1/library/itertools.html, "For instance, SML provides a tabulation tool: tabulate(f) which produces a sequence f(0), f(1), But, this effect can be achieved in Python by combining map() and count() to form map(f, count())." The second sentence should be "But this effect...", not "But, this effect". No issue is too small, right? :) -- assignee: georg.brandl components: Documentation messages: 98100 nosy: Justin.Lebar, georg.brandl severity: normal status: open title: Nit: Extra comma in itertools doc versions: Python 3.1 ___ Python tracker <http://bugs.python.org/issue7746> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7797] base64 module docs should indicate that encode methods return bytes, not strings
New submission from Justin Lebar : It's not at all clear from the documentation that base64.base64encode() and its kin return bytes, rather than strings. Since this matters now, the docs should be clear on this point. http://docs.python.org/3.1/library/base64.html -- assignee: georg.brandl components: Documentation messages: 98458 nosy: Justin.Lebar, georg.brandl severity: normal status: open title: base64 module docs should indicate that encode methods return bytes, not strings type: feature request versions: Python 3.1 ___ Python tracker <http://bugs.python.org/issue7797> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7809] Documentation for random module should indicate that a call to seed() is not required
New submission from Justin Lebar : Many programmers are used to languages where the RNG is deterministic unless it's explicitly seeded. This does not appear to be the case in Python. The documentation for random should indicate that Random objects are seeded on construction, and that the global random object is seeded when the random package is first imported. http://docs.python.org/3.1/library/random.html -- assignee: georg.brandl components: Documentation messages: 98551 nosy: Justin.Lebar, georg.brandl severity: normal status: open title: Documentation for random module should indicate that a call to seed() is not required ___ Python tracker <http://bugs.python.org/issue7809> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7809] Documentation for random module should indicate that a call to seed() is not required
Justin Lebar added the comment: Ack. I read it twice looking for that line! Perhaps it should still indicate that new Random objects are automatically seeded. And perhaps the line I missed should be a little bit louder? -- status: pending -> open ___ Python tracker <http://bugs.python.org/issue7809> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7864] Deprecation markers in unittest docs are unclear
New submission from Justin Lebar : The documentation for assertTrue/assert_/failUnless reads: assertTrue(expr, msg=None) assert_(expr, msg=None) failUnless(expr, msg=None) ... Deprecated since version 3.1: failUnless(). The deprecation warning is confusing, since it appears to suggest that failUnless is also deprecated. Perhaps the last line could read assertTrue and assert_ are deprecated since version 3.1: Use failUnless() instead. And similarly for the rest of the assertX methods. -- assignee: georg.brandl components: Documentation messages: 98933 nosy: Justin.Lebar, georg.brandl severity: normal status: open title: Deprecation markers in unittest docs are unclear versions: Python 3.1 ___ Python tracker <http://bugs.python.org/issue7864> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7864] Deprecation markers in unittest docs are unclear
Justin Lebar added the comment: Oh, I see. I still think a complete sentence would be helpful here, although now that I understand that the colon doesn't mean "use instead" I suppose I'm less confused... -- ___ Python tracker <http://bugs.python.org/issue7864> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7995] On Mac / BSD sockets returned by accept inherit the parent's FD flags
New submission from Justin Cappos : Suppose there is a program that has a listening socket that calls accept to obtain new sockets for client connections. socketmodule.c assumes that these client sockets have timeouts / blocking in the default state for new sockets (which on most systems means the sockets will block). However, socketmodule.c does not verify the state of the socket object that is returned by the system call accept. >From http://linux.die.net/man/2/accept : On Linux, the new socket returned by accept() does not inherit file status flags such as O_NONBLOCK and O_ASYNC from the listening socket. This behaviour differs from the canonical BSD sockets implementation. Portable programs should not rely on inheritance or non-inheritance of file status flags and always explicitly set all required flags on the socket returned from accept(). socketmodule.c does not explicitly set or check these flags for sockets returned by accept. The attached program will print the following on Linux regardless of whether the settimeout line for s exists or not: a has timeout: None O_NONBLOCK is set: False received: hi On Mac / BSD, the program will produce the following output when the timeout is set on the listening socket: a has timeout: None O_NONBLOCK is set: True Traceback (most recent call last): File "python-nonportable.py", line 39, in message = a.recv(1024) socket.error: (35, 'Resource temporarily unavailable') When the timeout is removed, the behavior is the same as linux: a has timeout: None O_NONBLOCK is set: False received: hi Note that the file descriptor problem crops up in odd ways on Mac systems. It's possible that issue 5154 may be due to this bug. I am aware of other problems with the socketmodule on Mac and will report them in other tickets. I believe that this problem can be easily mitigated by always calling fcntl to unset the O_NONBLOCK flag after accept (O_ASYNC should be unset too, for correctness). I would recommend adding the below code snippet at line 1653 in socketmodule.c (r78335). The resulting code would look something like this (with '+' in front of the added lines): ''' #ifdef MS_WINDOWS if (newfd == INVALID_SOCKET) #else if (newfd < 0) #endif return s->errorhandler(); +#if defined(__APPLE__) || defined(__OpenBSD__) || defined(__FreeBSD__) +int starting_flag; +// Unset O_NONBLOCK an O_ASYNC if they are inherited. +starting_flag = fcntl(newfd, F_GETFL, 0); +starting_flag &= ~(O_NONBLOCK | O_ASYNC); +fcntl(newfd, F_SETFL, starting_flag); +#endif /* Create the new object with unspecified family, to avoid calls to bind() etc. on it. */ sock = (PyObject *) new_sockobject(newfd, s->sock_family, s->sock_type, s->sock_proto); ''' I've tested this patch on my Mac and Linux systems and it seems to work fine. I haven't had a chance to test on BSD. Also, I did not test for this problem in Python 3, but I assume it exists there as well and the same fix should be applied. -- assignee: ronaldoussoren components: Library (Lib), Macintosh files: python-nonportable.py messages: 99852 nosy: Justin.Cappos, bbangert, giampaolo.rodola, loewis, nicdumz, ronaldoussoren severity: normal status: open title: On Mac / BSD sockets returned by accept inherit the parent's FD flags type: behavior versions: Python 2.5, Python 2.6, Python 2.7 Added file: http://bugs.python.org/file16325/python-nonportable.py ___ Python tracker <http://bugs.python.org/issue7995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7850] platform.system() should be "macosx" instead of "Darwin" on OSX
Justin Cappos added the comment: Perhaps the right way to fix the problem without breaking code would be to propose a new function for platform which would return a 'newbie readable' string of the system type? -- nosy: +Justin.Cappos ___ Python tracker <http://bugs.python.org/issue7850> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8637] [PATCH] pydoc should respect MANPAGER over PAGER.
New submission from Justin Bronder : Similar to man(1), pydoc should respect the environment variable MANPAGER over PAGER. The current behavior causes issues when using a PAGER that does not support man pages. Patch based off of trunk, r80871 -- components: Library (Lib) files: python-2.7-pydoc-manpager.patch keywords: patch messages: 105151 nosy: jsbronder priority: normal severity: normal status: open title: [PATCH] pydoc should respect MANPAGER over PAGER. type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file17238/python-2.7-pydoc-manpager.patch ___ Python tracker <http://bugs.python.org/issue8637> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8637] pydoc should respect MANPAGER over PAGER.
Justin Bronder added the comment: I chose MANPAGER as the command line pydoc utility already clones some of the man functionality, but I'm fine with PYDOCPAGER or anything else that allows me to use pydoc without having to change PAGER first. -- ___ Python tracker <http://bugs.python.org/issue8637> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2586] Integer signedness bugs in zlib modules
New submission from Justin Ferguson <[EMAIL PROTECTED]>: The zlib module in multiple places fails to adequately check the sanity of its arguments resulting in memory corruption, please see two attached PoCs. -- components: Extension Modules files: python-2.5.2-zlib-unflush-misallocation.py messages: 65171 nosy: jnferguson severity: normal status: open title: Integer signedness bugs in zlib modules type: security versions: Python 2.5 Added file: http://bugs.python.org/file9983/python-2.5.2-zlib-unflush-misallocation.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2586> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2586] Integer signedness bugs in zlib modules
Changes by Justin Ferguson <[EMAIL PROTECTED]>: Added file: http://bugs.python.org/file9984/python-2.5.2-zlib-unflush-signedness.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2586> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsane
New submission from Justin Ferguson <[EMAIL PROTECTED]>: The PyString_FromStringAndSize() function takes a pointer and signed integer as input parameters however it fails to adequately check the sanity of the integer argument. Because of the failure to check for negative values and because it sums the integer with the size of the PyStringObject structure it becomes possible for the allocator to take either of the code paths in PyObject_MALLOC()-- both of which will incorrectly allocate memory. This may not seem like a big deal, but I'm posting this instead of filing a bug for every place this screws you guys over. if (0 > len || len > PYSSIZE_T_MAX/sizeof(PyStringObject)) return NULL; -- components: Interpreter Core messages: 65172 nosy: jnferguson severity: normal status: open title: PyString_FromStringAndSize() to be considered unsane type: security versions: Python 2.5 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
The PyOS_vsnprintf() contains the caveat that the length parameter
cannot be zero, however this is only enforced via assert() which is
compiled out. As a result if the length parameter is zero then the
function will underflow and write a null byte to invalid memory.
53 int
54 PyOS_vsnprintf(char *str, size_t size, const char *format, va_list va)
55 {
56 int len; /* # bytes written, excluding \0 */
57 #ifndef HAVE_SNPRINTF
58 char *buffer;
59 #endif
60 assert(str != NULL);
61 assert(size > 0);
62 assert(format != NULL);
[...]
65 len = vsnprintf(str, size, format, va);
[...]
91 str[size-1] = '\0';
92 return len;
93 }
--
components: Distutils
messages: 65174
nosy: jnferguson
severity: normal
status: open
title: PyOS_vsnprintf() underflow leads to memory corruption
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2588>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
Changes by Justin Ferguson <[EMAIL PROTECTED]>: -- components: +Interpreter Core -Distutils __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2588> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2589] PyOS_vsnprintf() potential integer overflow leads to memory corruption on esoteric architectures
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
On architectures that do not have a vsnprintf() in their standard
library Python attempts to emulate it. When doing so, the implementation
ambitiously allocates more memory than requested without verifying the
sanity of the summed value. As a result it becomes possible (although
unlikely) for an integer overflow to occur misallocating memory and
causing a buffer overflow.
53 int
54 PyOS_vsnprintf(char *str, size_t size, const char *format, va_list va)
55 {
56 int len; /* # bytes written, excluding \0 */
[...]
60 assert(str != NULL);
61 assert(size > 0);
62 assert(format != NULL);
63
[...]
67 /* Emulate it. */
68 buffer = PyMem_MALLOC(size + 512);
69 if (buffer == NULL) {
70 len = -666;
71 goto Done;
72 }
73
74 len = vsprintf(buffer, format, va);
75 if (len < 0)
76 /* ignore the error */;
77
78 else if ((size_t)len >= size + 512)
79 Py_FatalError("Buffer overflow in
PyOS_snprintf/PyOS_vsnprintf");
80
81 else {
82 const size_t to_copy = (size_t)len < size ?
83 (size_t)len : size - 1;
84 assert(to_copy < size);
85 memcpy(str, buffer, to_copy);
86 str[to_copy] = '\0';
87 }
88 PyMem_FREE(buffer);
89 Done:
[...]
91 str[size-1] = '\0';
92 return len;
93 }
--
components: Interpreter Core
messages: 65175
nosy: jnferguson
severity: normal
status: open
title: PyOS_vsnprintf() potential integer overflow leads to memory corruption
on esoteric architectures
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2589>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2590] S_unpack_from() Read Access Violation
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
The S_unpack_from() function in Modules/_struct.c does not adequately
validate its arguments, potentially causing an out-of-bounds read
access. It should be noted that the check at line 1561 is inadequate for
obscene values of offset. Finally, because they're not really important
and I really don't want to type them all up-- you guys might want to go
through your code-- especially the modules and look for constructs where
an empty string will cause memory to be uninitialized-- look at the
audioop module for examples of what I mean-- the only thing that
actually saved you guys from overflows there was that the loops you
write with use the same variable.
1533 static PyObject *
1534 s_unpack_from(PyObject *self, PyObject *args, PyObject *kwds)
1535 {
1536 static char *kwlist[] = {"buffer", "offset", 0};
1537 #if (PY_VERSION_HEX < 0x0205)
1538 static char *fmt = "z#|i:unpack_from";
1539 #else
1540 static char *fmt = "z#|n:unpack_from";
1541 #endif
1542 Py_ssize_t buffer_len = 0, offset = 0;
[...]
1547
1548 if (!PyArg_ParseTupleAndKeywords(args, kwds, fmt, kwlist,
1549 &buffer, &buffer_len,
&offset))
1550 return NULL;
[...]
1558 if (offset < 0)
1559 offset += buffer_len;
1560
1561 if (offset < 0 || (buffer_len - offset) < soself->s_size) {
[...]
1566 }
1567 return s_unpack_internal(soself, buffer + offset);
1568 }
--
components: Extension Modules
messages: 65178
nosy: jnferguson
severity: normal
status: open
title: S_unpack_from() Read Access Violation
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2590>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2591] ErrorHandler buffer overflow in ?unused? SGI extension module almodule.c
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
I don't think any of these SGI modules even get used, but they're really
buggy-- you guys might want to consider just dropping them all together.
When printing errors larger than 128 bytes a stack based overflow occurs.
44 static void
45 ErrorHandler(long code, const char *fmt, ...)
46 {
47 va_list args;
48 char buf[128];
49
50 va_start(args, fmt);
51 vsprintf(buf, fmt, args);
52 va_end(args);
53 PyErr_SetString(ErrorObject, buf);
54 }
--
components: Extension Modules
messages: 65180
nosy: jnferguson
severity: normal
status: open
title: ErrorHandler buffer overflow in ?unused? SGI extension module almodule.c
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2591>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2593] alp_ReadFrames() integer overflow leads to buffer overflow
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
Please see bug 2591 for a suggestion on what to do with these SGI modules.
(sorry I don't have any pocs/repros I dont have an sgi box handy atm)
Integer overflow/invalid allocation at 768, write to memory at 773
716 static PyObject *
717 alp_ReadFrames(alpobject *self, PyObject *args)
718 {
719 int framecount;
720 PyObject *v;
721 int size;
722 int ch;
723 ALconfig c;
724
725 if (!PyArg_ParseTuple(args, "i:ReadFrames", &framecount))
726 return NULL;
727 if (framecount < 0) {
728 PyErr_SetString(ErrorObject, "negative framecount");
729 return NULL;
730 }
[...] 732 switch (alGetSampFmt(c)) {
733 case AL_SAMPFMT_TWOSCOMP:
734 switch (alGetWidth(c)) {
735 case AL_SAMPLE_8:
736 size = 1;
737 break;
738 case AL_SAMPLE_16:
739 size = 2;
740 break;
741 case AL_SAMPLE_24:
742 size = 4;
743 break;
744 default:
745 PyErr_SetString(ErrorObject, "can't
determine width");
746 alFreeConfig(c);
747 return NULL;
748 }
749 break;
750 case AL_SAMPFMT_FLOAT:
751 size = 4;
752 break;
753 case AL_SAMPFMT_DOUBLE:
754 size = 8;
755 break;
756 default:
757 PyErr_SetString(ErrorObject, "can't determine format");
758 alFreeConfig(c);
759 return NULL;
760 }
761 ch = alGetChannels(c);
762 alFreeConfig(c);
763 if (ch < 0) {
764 PyErr_SetString(ErrorObject, "can't determine # of
channels");
765 return NULL;
766 }
767 size *= ch;
768 v = PyString_FromStringAndSize((char *) NULL, size *
framecount);
769 if (v == NULL)
770 return NULL;
771
[...]
773 alReadFrames(self->port, (void *) PyString_AS_STRING(v),
framecount);
--
components: Extension Modules
messages: 65183
nosy: jnferguson
severity: normal
status: open
title: alp_ReadFrames() integer overflow leads to buffer overflow
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2593>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsane
Justin Ferguson <[EMAIL PROTECTED]> added the comment: The problem with assert()'s is they require debugging to be enabled, obviously, who compiles it that way? You may not even want to worry about the second check, when you pass it into the allocator it gets converted to an unsigned int which will cause the allocator to either fail (32-bit) or allocate more memory than expected-- either cause it handled/benign. If you'd prefer I can file every place where this actually bites you guys, I was just trying to be nice. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
Justin Ferguson <[EMAIL PROTECTED]> added the comment: I can generally agree with that, and I admit I haven't verified all of the code paths here- theres several hundred of them leading into this function, are you positive all of them are safe? (seems like it would be easier to just move the check into an if than sitting down and verifying that all XXX hundred code paths are safe). In the other bug, I have verified code paths into it, for instance test the misallocation poc in 2586 as an example __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2588> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsane
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Adding a poc from 2586 to demonstrate my point, this causes a call to the allocator requesting zero bytes. Added file: http://bugs.python.org/file9985/python-2.5.2-zlib-unflush-misallocation.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2594] alp_readsamps() overflow leads to memory corruption in ?unused? SGI extension module almodule.c
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
You guys should probably just remove the SGI modules, the code looks
like it hasn't been touched in some time and hasn't gone through the
same security checks as other pieces of code. Sorry I have no
repro's/pocs, I don't have an irix box either though ;]
integer overflow/misallocation occurs at 1071, write to bad memory at 1076
1042 alp_readsamps(alpobject *self, PyObject *args)
1043 {
1044 long count;
1045 PyObject *v;
1046 ALconfig c;
1047 int width;
1048 int ret;
1049
1050 if (!PyArg_ParseTuple(args, "l:readsamps", &count))
1051 return NULL;
1052
1053 if (count <= 0) {
1054 PyErr_SetString(ErrorObject, "al.readsamps : arg <=
0");
1055 return NULL;
1056 }
1057
1058 c = ALgetconfig(self->port);
1059 #ifdef AL_405
1060 width = ALgetsampfmt(c);
1061 if (width == AL_SAMPFMT_FLOAT)
1062 width = sizeof(float);
1063 else if (width == AL_SAMPFMT_DOUBLE)
1064 width = sizeof(double);
1065 else
1066 width = ALgetwidth(c);
1067 #else
1068 width = ALgetwidth(c);
1069 #endif /* AL_405 */
1070 ALfreeconfig(c);
1071 v = PyString_FromStringAndSize((char *)NULL, width * count);
1072 if (v == NULL)
1073 return NULL;
1074
1075 Py_BEGIN_ALLOW_THREADS
1076 ret = ALreadsamps(self->port, (void *)
PyString_AsString(v), count);
1077 Py_END_ALLOW_THREADS
1078 if (ret == -1) {
1079 Py_DECREF(v);
1080 return NULL;
1081 }
1082
1083 return (v);
1084 }
--
components: Extension Modules
messages: 65188
nosy: jnferguson
severity: normal
status: open
title: alp_readsamps() overflow leads to memory corruption in ?unused? SGI
extension module almodule.c
type: security
versions: Python 2.5
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2594>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2586] Integer signedness bugs in zlib modules
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Just so you know, the scripts actually do two different things-- the bugs are both related to negative values though. One causes PyString_FromStringAndSize() to try an allocate zero bytes (the -24 one), the other causes like 22 bytes to get allocated and then takes advantage of the sign-conversion when the value is assigned to the zlib structure member (the member is unsigned, the value is signed) Honestly, you guys should consider enforcing the safe downcast usage because signedness issues are all over your code base (as I'm sure you know) __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2586> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2595] Multiple integer overflows in imgfile extension module lead to buffer overflow
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
The imgfile module contains multiple integer overflows, this module is
only used on SGI boxes and is likely mostly unused and thus is fairly
low priority imho-- no repros, no poc, no sgi box :/
I'm only going to post one to give you the idea, there's no need for me
to (further) spam the bug database by filing a bug for each one of
these, they're all pretty much the same.
Here the variables xsize, ysize and zsize are all externally derived.
While xsize and zsize are sanity checked, ysize is not. This potentially
results in an integer overflow/misallocation at line 133 and writes to
invalid memory in the calls to getrow()
85 static PyObject *
86 imgfile_read(PyObject *self, PyObject *args)
87 {
88 char *fname;
89 PyObject *rv;
90 int xsize, ysize, zsize;
91 char *cdatap;
92 long *idatap;
93 static short rs[8192], gs[8192], bs[8192];
94 int x, y;
95 IMAGE *image;
96 int yfirst, ylast, ystep;
97
98 if ( !PyArg_ParseTuple(args, "s:read", &fname) )
99 return NULL;
100
101 if ( (image = imgfile_open(fname)) == NULL )
102 return NULL;
[...]
116 xsize = image->xsize;
117 ysize = image->ysize;
118 zsize = image->zsize;
119 if ( zsize != 1 && zsize != 3) {
120 iclose(image);
121 PyErr_SetString(ImgfileError,
122 "Can only handle 1 or 3 byte pixels");
123 return NULL;
124 }
125 if ( xsize > 8192 ) {
126 iclose(image);
127 PyErr_SetString(ImgfileError,
128 "Can't handle image with > 8192
columns");
129 return NULL;
130 }
131
132 if ( zsize == 3 ) zsize = 4;
133 rv = PyString_FromStringAndSize((char *)NULL,
xsize*ysize*zsize);
134 if ( rv == NULL ) {
138 cdatap = PyString_AsString(rv);
139 idatap = (long *)cdatap;
[...]
150 for ( y=yfirst; y != ylast && !error_called; y += ystep ) {
151 if ( zsize == 1 ) {
152 getrow(image, rs, y, 0);
153 for(x=0; x
<http://bugs.python.org/issue2595>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2595] Multiple integer overflows in imgfile extension module lead to buffer overflow
Justin Ferguson <[EMAIL PROTECTED]> added the comment: I'm not going to file a different bug for rgbimgmodule.c, it has the same types of issues. Sorry, I don't mean to drop a ton of bugs, I'm prepping up to do a talk on attacking the metadata in scripting languages (i.e. the python call stack instead of the processors) and I need to have these public/patched before I talk about them. I've got a bunch more bugs, I'll file them later when some of this stuff is caught up with. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2595> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsane
Justin Ferguson <[EMAIL PROTECTED]> added the comment:
As an addemdum, consider the following code (theres no assert, but it
wouldnt have helped you outside of debug builds anyways):
488 static PyObject *PySSL_SSLread(PySSLObject *self, PyObject *args)
489 {
490 PyObject *buf;
491 int count = 0;
492 int len = 1024;
[...]
496 if (!PyArg_ParseTuple(args, "|i:read", &len))
497 return NULL;
498
499 if (!(buf = PyString_FromStringAndSize((char *) 0, len)))
500 return NULL;
[...]
521 count = SSL_read(self->ssl, PyString_AsString(buf),
len);
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2587>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
Justin Ferguson <[EMAIL PROTECTED]> added the comment: I do agree with your point about snprintf(..., sizeof(x), ...)-- my single biggest point (and maybe i'm just not seeing it), is that there appears to be no good reason for having this caveat and in turn its essentially just code waiting to break; with as commonly used of a function as it is, it's really a matter of when and not so much if. While no one seems to ever use it this way, don't forget that a good alternative to asprintf() is calling sprintf() with a length of zero to get the length (in compliant implementations), allocating the memory and then calling it again. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2588> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsafe
Justin Ferguson <[EMAIL PROTECTED]> added the comment: While I think its a mistake that will cause security concerns in Python for quite some time to come, thats fair enough. I will refile all the places that this messes up individually __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Actually, I'm not sure things are any better today- even the same operating system but different versions have inconsistencies, for instance on some versions of RHEL the vsnprintf() can fail during unicode conversion. MSVCRT still returns -1 on truncation, et cetera. That said, theres plenty of other implementations that manage this without the potential of underflowing a buffer __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2588> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsafe
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Do I need to create proof of concepts for each of these bugs, or can I reference this ticket? __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2588] PyOS_vsnprintf() underflow leads to memory corruption
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Yep, that works for me. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2588> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsafe
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Okay, so I'm not sure whose point of view takes precedence here? Also, to answer the question asked, I'm doing this manually. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsafe
Justin Ferguson <[EMAIL PROTECTED]> added the comment: > 1. If you change PyString_FromStringAndSize, you should change > PyBytes_FromStringAndSize and PyUnicode_FromStringAndSize for > consistency. Okay, if we don't patch the API call itself, I will look at the users of those two API calls as well, I haven't gone through much of the stuff in Objects/ yet, I focused on the modules obviously and thus didn't know about those calls. > 4. You have clearly identified several bugs that are not covered by > unittests. If you fix them en-mass by making > PyString_FromStringAndSize more forgiving, we will miss an opportunity > to improve the test coverage. (If you take the SystemError approach, > this does not apply.) This is a valid and good point-- I don't mind either way __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2587> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2587] PyString_FromStringAndSize() to be considered unsafe
Justin Ferguson <[EMAIL PROTECTED]> added the comment:
The use of signed integers in Python is (imho) the single largest threat
to the security of the interpreter. I'm probably preaching to the choir
there though.
I really dislike have to return values and indicate error in the return
value, its really unclean and causes weirdness, for instance if you
follow PyArg_ParseTuple() down and you have an integer (or maybe it was
long?) argument, you can't actually get a value of -1 because one of the
string->int conversion routines uses that to indicate failure. The check
wrapped around it was something along the lines of:
if (-1 == retval && PyErr_Occurred())
In turn down the line somewhere (I didn't follow the code path), this
value got converted to 1, so for instance doing
__import__('zlib').decompressobj().flush(-1) wouldn't trigger that bug,
it would flush 1 byte, not UINT_MAX as I had expected
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2587>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
New submission from Justin Ferguson <[EMAIL PROTECTED]>:
174 static
175 int unicode_resize(register PyUnicodeObject *unicode,
176 Py_ssize_t length)
177 {
[...]
201
202 oldstr = unicode->str;
203 PyMem_RESIZE(unicode->str, Py_UNICODE, length + 1);
[...]
209 unicode->str[length] = 0;
210 unicode->length = length;
211
95 #define PyMem_RESIZE(p, type, n) \
96 ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
97 ( (p) = (type *) PyMem_REALLOC((p), (n) * sizeof(type)) ) )
The unicode_resize() function acts essentially as a wrapper to
realloc(), it accomplishes this via the PyMem_RESIZE() macro which
factors the size with the size of the type, in this case it multiplies
by two as Py_UNICODE is typedef'd to a wchar_t. When resizing large
strings, this results in an incorrect allocation that in turn leads to
buffer overflow.
This is specific to the Unicode objects, however I would not be
surprised to see that other types have this complication as well. Please
see attached proof of concepts.
--
components: Interpreter Core
files: python-2.5.2-unicode_resize-utf7.py
messages: 65379
nosy: jnferguson
severity: normal
status: open
title: Multiple buffer overflows in unicode processing
type: security
versions: Python 2.5
Added file: http://bugs.python.org/file10011/python-2.5.2-unicode_resize-utf7.py
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2620>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Changes by Justin Ferguson <[EMAIL PROTECTED]>: Added file: http://bugs.python.org/file10012/python-2.5.2-unicode_resize-utf8.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Changes by Justin Ferguson <[EMAIL PROTECTED]>: Added file: http://bugs.python.org/file10013/python-2.5.2-unicode_resize-utf16.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2590] S_unpack_from() Read Access Violation
Justin Ferguson <[EMAIL PROTECTED]> added the comment: What I was originally thinking was if offset was larger than buf_len, that would cause the check at 1561 to fail due to the subtraction. That said, I'm not sure what type its being compared against so I need to check this further, let me get back to you, I may have made a mistake and this may be !bug. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2590> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Justin Ferguson <[EMAIL PROTECTED]> added the comment: Yes, excuse me-- this should be 32-bit specific as I believe Python will not let me get a string long enough to overflow the integer on 64-bit. It's a big string, the only realistic scenario I can see is XML parsing or similar. theory$ ./python -V Python 2.5.2 theory$ cat /proc/cpuinfo | grep -i model model : 4 model name : Intel(R) Pentium(R) 4 CPU 3.00GHz theory$ ./python python-2.5.2-unicode_resize-utf7.py Segmentation fault __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Justin Ferguson <[EMAIL PROTECTED]> added the comment: i pulled the Macros out of pymem.h in a Vanille 2.5.2? -- components: +None versions: +Python 3.0 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Justin Ferguson <[EMAIL PROTECTED]> added the comment: sorry didnt mean to change components and version-- I'm typing this from my phone and its being uncooperative at the moment -- components: +Library (Lib) __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Justin Ferguson <[EMAIL PROTECTED]> added the comment: just fixing the modifications my phone made earlier tonight -- components: -Library (Lib), None versions: -Python 3.0 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2620> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2620] Multiple buffer overflows in unicode processing
Justin Ferguson <[EMAIL PROTECTED]> added the comment:
Additionally-- the PyMem_NEW()/PyMem_New() macro's need to be fixed:
231 static
232 PyUnicodeObject *_PyUnicode_New(Py_ssize_t length)
233 {
234 register PyUnicodeObject *unicode;
235
236 /* Optimization for empty strings */
237 if (length == 0 && unicode_empty != NULL) {
238 Py_INCREF(unicode_empty);
239 return unicode_empty;
240 }
241
242 /* Unicode freelist & memory allocation */
243 if (unicode_freelist) {
244 unicode = unicode_freelist;
245 unicode_freelist = *(PyUnicodeObject **)unicode;
246 unicode_freelist_size--;
247 if (unicode->str) {
248 /* Keep-Alive optimization: we only upsize the buffer,
249never downsize it. */
250 if ((unicode->length < length) &&
251 unicode_resize(unicode, length) < 0) {
252 PyMem_DEL(unicode->str);
253 goto onError;
254 }
255 }
256 else {
257 unicode->str = PyMem_NEW(Py_UNICODE, length + 1);
258 }
85 #define PyMem_New(type, n) \
86 ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
87 ( (type *) PyMem_Malloc((n) * sizeof(type)) ) )
88 #define PyMem_NEW(type, n) \
89 ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
90 ( (type *) PyMem_MALLOC((n) * sizeof(type)) ) )
91
92 #define PyMem_Resize(p, type, n) \
93 ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
94 ( (p) = (type *) PyMem_Realloc((p), (n) * sizeof(type)) ) )
95 #define PyMem_RESIZE(p, type, n) \
96 ( assert((n) <= PY_SIZE_MAX / sizeof(type)) , \
97 ( (p) = (type *) PyMem_REALLOC((p), (n) * sizeof(type)) ) )
It looks like much of this is reachable from modules, unicode objects,
dict objects, set objects, et cetera. Also, if a 2G string across the
network seems unrealistic consider what 2 billion A's compresses down
to.(Macros again taken from 2.5.2 vanilla)
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2620>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks
New submission from Justin ARthur :
Python 3 code with an identifier that has a non-spacing mark in it does not get
tokenized by lib2to3 and will result in an exception thrown in the parsing
process.
Parsing the attached file (badvar.py), results in `ParseError: bad token:
type=58, value='̇', context=('', (1, 1))`
This happens because the Name pattern regular expression in lib2to3 is `r'\w+'`
and the word character class doesn't contain non-spacing marks (and possible
other [continuation characters allowed in Python 3
identifiers](https://docs.python.org/3/reference/lexical_analysis.html#identifiers)).
(reported by energizer in the Python IRC channel)
--
components: 2to3 (2.x to 3.x conversion tool), Library (Lib)
files: badvar.py
messages: 351153
nosy: JustinTArthur
priority: normal
severity: normal
status: open
title: lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48592/badvar.py
___
Python tracker
<https://bugs.python.org/issue38032>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks
Justin Arthur added the comment: Ned, can you confirm that 2to3 is not intended for cumulative/incremental runs over the same codebase? If it's not intended to be run on previously ported code, this will just need to be fixed on the lib2to3 downstream projects like awpa and Black that are encountering this issue. -- ___ Python tracker <https://bugs.python.org/issue38032> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12731] python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a
Change by Justin Arthur : -- nosy: +JustinTArthur ___ Python tracker <https://bugs.python.org/issue12731> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38166] ast identifies incorrect column for compound attribute lookups
New submission from Justin McCann :
This issue is related to https://github.com/microsoft/vscode-python/issues/7327
and https://github.com/PyCQA/pylint/issues/3103
For compound attribute access in variable references like a.b.c.d, the AST
reports the column of the first variable/attribute in the sequence instead of
the specific attribute location. For example, the location of c is reported as
column 0 (a) and not column 4 (c).
Here's the AST test case provided by a pylint developer in a comment on pylint
issue 3103; I confirmed the behavior is the same in python 3.8.0b4.
```
$ python3.8
Python 3.8.0b4 (v3.8.0b4:d93605de72, Aug 29 2019, 21:47:47)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> body = ast.parse('''
... print(x.item.akey > 2)
... ''')
>>> # x.item
... x_item = body.body[0].value.args[0].left.value
>>> print(x_item, x_item.attr, x_item.col_offset)
<_ast.Attribute object at 0x10a7751f0> item 6
>>> # x.item.akey
... print(x_item.value, x_item.value.col_offset) . ### probably should be 8
<_ast.Name object at 0x10a775280> 6
```
Related issues:
* https://bugs.python.org/issue1440601 Add col information to parse & ast nodes
* https://bugs.python.org/issue10769 ast: provide more useful range information
Here is the resulting confusion when you use this output in pylint (and then
VSCode highlights only "x" since it's the variable that starts in column 0):
Original pylint/vscode testcase:
```
class TestMe:
def __init__(self):
self.item = {'akey': 42}
self.again = self
x = TestMe()
### pylint error message here is
###testme.py:11:6: E1101: Instance of 'dict' has no 'akey' member
(no-member)
### The problem is with `x.item`, but pylint shows the column for `x`
print(x.item.akey > 2)
print(x.again.item.doesnotexist)
```
Current behavior
$ pylint testme.py -rn -sn
* Module testme
testme.py:10:6: E1101: Instance of 'dict' has no 'akey' member (no-member)
testme.py:12:6: E1101: Instance of 'dict' has no 'doesnotexist' member
(no-member)
Expected behavior
$ pylint testme.py -rn -sn
* Module testme
testme.py:10:8: E1101: Instance of 'dict' has no 'akey' member (no-member)
testme.py:12:14: E1101: Instance of 'dict' has no 'doesnotexist' member
(no-member)
$ pylint --version output
pylint 2.3.1
astroid 2.2.5
Python 3.7.4 (default, Jul 9 2019, 18:13:23)
[Clang 10.0.1 (clang-1001.0.46.4)]
--
components: Library (Lib)
messages: 352393
nosy: Justin McCann
priority: normal
severity: normal
status: open
title: ast identifies incorrect column for compound attribute lookups
type: behavior
versions: Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue38166>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
