[issue25945] Type confusion in partial_setstate and partial_call leads to memory corruption

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee: serhiy.storchaka -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25455] Some repr implementations don't check for self-referential structures

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25822] Add docstrings to fields of urllib.parse results

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7bfcb8b75ad9 by Senthil Kumaran in branch 'default':
Issue #25822: Add docstrings to the fields of urllib.parse results.
https://hg.python.org/cpython/rev/7bfcb8b75ad9

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25822] Add docstrings to fields of urllib.parse results

[Senthil Kumaran]

Senthil Kumaran added the comment:

Thank you, Swati. 

I reviewed and committed your patch. I had to make some minor formatting 
changes while keeping an eye for the maintainability of the module and doc 
strings.

* Moved all the doc strings up to a single place just below the namespaces are 
declared.
* Followed pep-0257 conventions for doc strings.
* Tried DRY for parse result's sub-components.

Thanks for addressing the review comments from other developers.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26106] Move licences to literal blocks

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Does the |release| substitution work in literal blocks?

--
nosy: +brett.cannon, georg.brandl, ned.deily, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[STINNER Victor]

New submission from STINNER Victor:

Python doesn't store the original line number in the .pyc file in the bytecode. 
Instead, an efficient table is used to find the line number from the current in 
the bytecode: code.co_lnotab.

Basically, it's a list of (offset_delta, line_number_delta) pairs where 
offset_delta and line_number_delta are unsigned 8 bits numbers. If an offset 
delta is larger than 255, (offset_delta % 255, line_number_delta) and 
(offset_delta // 255, 0) pairs are emited. Same for line_number_delta. (In 
fact, more than two pairs can be created.)

The format is described in Objects/lnotab_notes.txt.

I implemented an optimizer which can generate *negative* line number. For 
example, the loop:

   for i in range(2):   # line 1
  print(i)  # line 2

is replaced with:

   i = 0  # line 1
   print(i)   # line 2
   i = 1  # line 1
   print(i)   # line 2

The third instruction has a negative line number delta.

I'm not the first one hitting the issue, but it's just that no one proposed a 
patch before. Previous projects bitten by this issue:

* issue #10399: "AST Optimization: inlining of function calls"
* issue #11549: "Build-out an AST optimizer, moving some functionality out of 
the peephole optimizer"

Attached patch changes the type of line number delta from unsigned 8-bit 
integer to *signed* 8-bit integer. If a line number delta is smaller than -128 
or larger than 127, multiple pairs are created (as before).

My code in Lib/dis.py is inefficient. Maybe unpack the full lnotab than *then* 
skip half of the bytes? (instead of calling struct.unpack times for each byte).

The patch adds also "assert(Py_REFCNT(lnotab_obj) == 1);" to PyCode_Optimize(). 
The assertion never fails, but it's just to be extra safe.

The patch renames variables in PyCode_Optimize() because I was confused between 
"offset" and "line numbers". IMHO variables were badly named.

I changed the MAGIC_NUMBER of importlib, but it was already changed for 
f-string:

# Python 3.6a0  3360 (add FORMAT_VALUE opcode #25483)

Is it worth to modify it again?

You may have to recompile Python/importlib_external.h if it's not recompiled 
automatically (just touch the file before running make).

Note: this issue is related to the PEP 511 (the PEP is not ready for a review, 
but it gives a better overview of the use cases.)

--
files: lnotab.patch
keywords: patch
messages: 258189
nosy: brett.cannon, haypo, rhettinger, serhiy.storchaka
priority: normal
severity: normal
status: open
title: code.co_lnotab: use signed line number delta to support moving 
instructions in an optimizer
versions: Python 3.6
Added file: http://bugs.python.org/file41613/lnotab.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[STINNER Victor]

Changes by STINNER Victor :


--
components: +Interpreter Core
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[STINNER Victor]

STINNER Victor added the comment:

> Attached patch changes the type of line number delta from unsigned 8-bit 
> integer to *signed* 8-bit integer. If a line number delta is smaller than 
> -128 or larger than 127, multiple pairs are created (as before).

The main visible change is that .pyc files can be a little bit larger and so 
use more disk space. Well... in fact only .pyc of files using line number delta 
larger than 127.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[Roundup Robot]

Roundup Robot added the comment:

New changeset c4a826184937 by Victor Stinner in branch 'default':
PEP 511: add link to issue #26107
https://hg.python.org/peps/rev/c4a826184937

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26106] Move licences to literal blocks

[Julien]

Julien added the comment:

@Serhiy Well spotted, it does not work (as we can see here 
http://www.afpy.org/doc/python/3.5/license.html).

Anyone having a better background than me in ReStructuredText have an idea ?

About the review of the patch, as a human review may be fastidious, here a 
script to check that I did NOT modified the content of the licences themselves 
(still need to trust the command, but it's faster to review than the whole 
licence text itself):

diff -w <(grep '^-' literal-licence.patch | cut -b2-) <(grep '^+' 
literal-licence.patch | cut -b2- | sed 's/\(^ *\)[0-8]\./\1#./g')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26106] Move licences to literal blocks

[Georg Brandl]

Georg Brandl added the comment:

Since the text does need no highlighting, a parsed-literal block should work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26108] Calling PyInitialize with 2.7.11 on Windows x64 terminates process

[David Heffernan]

New submission from David Heffernan:

Environment: 
 - Python 2.7.11 from python.org, x64.
 - Windows 10 or Windows 8.1
 - MSVC 2015

I compiled the most basic embedding example, taken from the Python docs:

#include 

int
main(int argc, char *argv[])
{
  Py_SetProgramName(argv[0]);  /* optional but recommended */
  Py_Initialize();
  PyRun_SimpleString("from time import time,ctime\n"
 "print 'Today is',ctime(time())\n");
  Py_Finalize();
  return 0;
}

When run the call to Py_Initialize does not return and the process is 
terminated.

--
components: Windows
messages: 258194
nosy: David Heffernan, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Calling PyInitialize with 2.7.11 on Windows x64 terminates process
type: crash
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26108] Calling PyInitialize with 2.7.11 on Windows x64 terminates process

[David Heffernan]

David Heffernan added the comment:

Note that I've just listed the Windows versions on which I have tested this. I 
have not tested on Windows 7 or Vista so do not know whether or not the issue 
exists there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25945] Type confusion in partial_setstate and partial_call leads to memory corruption

[Nick Coghlan]

Nick Coghlan added the comment:

I didn't do a full review of the C code changes, but the new test cases look 
good to me, and the changes specifically to partial_setstate also look good.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

A patch was proposed in issue16956. And issue17611 is related.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26108] Calling PyInitialize with 2.7.11 on Windows x64 terminates process

[Eryk Sun]

Eryk Sun added the comment:

Probably this is related to issue 25824. When built as a console application 
and run from the command prompt, you should see an the error

ImportError: No module named site

If that's the case, the problem is that even the 64-bit build is incorrectly 
setting the DLL version string to "2.7-32":

0:000> da poi(python27!PyWin_DLLVersionString)
`50b878c0  "2.7-32"

So the interpreter is looking at the wrong registry key to get the default 
sys.path:

python27!getpythonregpath+0x110:
`509a3d90 ff15b2d20500callqword ptr
[python27!_imp_RegOpenKeyExA (`50a01048)]
ds:`50a01048=
{ADVAPI32!RegOpenKeyExAStub (7ffb`56f87d70)}
0:000> da @rdx
0037`e4fd9940  "Software\Python\PythonCore\2.7-3"
0037`e4fd9960  "2\PythonPath"

As a workaround, before calling Py_Initialize, add 
Py_SetPythonHome("C:\\Python27"), or wherever you installed Python. Or create a 
symbolic link to Python's Lib directory in the directory that has your 
executable:

mklink /d Lib "C:\Python27\Lib"

Or 'fix' the "SOFTWARE\Python\PythonCore\2.7" registry key by renaming it to 
"2.7-32". Or downgrade to 2.7.10 until 2.7.12 is released. 

That said, probably you'll use a zipped library if distributing an application, 
in which case this shouldn't be a problem.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26109] _Py_DumpTraceback should be PyAPI_FUNC

[John Malmberg]

New submission from John Malmberg:

The _PyDumpTraceback and _Py_DumpTracebackThreads routines in traceback.h are 
tagged with PyAPI_DATA attributes when they should be tagged with PyAPI_FUNC.

For platforms that use those attributes, this can cause run-time issues if 
those methods are called.

--
components: Interpreter Core
messages: 258199
nosy: John.Malmberg
priority: normal
severity: normal
status: open
title: _Py_DumpTraceback should be PyAPI_FUNC
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25940] SSL tests failed due to expired svn.python.org SSL certificate

[Roundup Robot]

Roundup Robot added the comment:

New changeset 81b3beea7e99 by Georg Brandl in branch '3.2':
Issue #25940: Backport self-signed.pythontest.net testing for test_httplib
https://hg.python.org/cpython/rev/81b3beea7e99

New changeset adf750b1252d by Martin Panter in branch '3.2':
Issue #25940: Use self-signed.pythontest.net in SSL tests
https://hg.python.org/cpython/rev/adf750b1252d

New changeset e167a9296947 by Martin Panter  in branch '3.3':
Issue #25940: Merge self-signed.pythontest.net testing from 3.2 into 3.3
https://hg.python.org/cpython/rev/e167a9296947

New changeset 32bcb9aa286e by Martin Panter  in branch '3.4':
Issue #25940: Merge self-signed.pythontest.net testing from 3.3 into 3.4
https://hg.python.org/cpython/rev/32bcb9aa286e

New changeset 0585c0089466 by Martin Panter  in branch '3.4':
Issue #25940: Update new SSL tests for self-signed.pythontest.net
https://hg.python.org/cpython/rev/0585c0089466

New changeset c3aa4b48b905 by Martin Panter  in branch '3.5':
Issue #25940: Merge self-signed.pythontest.net testing from 3.4 into 3.5
https://hg.python.org/cpython/rev/c3aa4b48b905

New changeset f5f14d65297e by Martin Panter  in branch '3.5':
Issue #25940: Update new SSL tests for self-signed.pythontest.net
https://hg.python.org/cpython/rev/f5f14d65297e

New changeset 2f2b42a8b34d by Martin Panter  in branch 'default':
Issue #25940: Merge self-signed.pythontest.net testing from 3.5
https://hg.python.org/cpython/rev/2f2b42a8b34d

New changeset ac94418299bd by Martin Panter  in branch 'default':
Issue #25940: test_ssl is working again
https://hg.python.org/cpython/rev/ac94418299bd

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26109] _Py_DumpTraceback should be PyAPI_FUNC

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Correction for searching: _Py_DumpTraceback and _Py_DumpTracebackThreads. They 
were added with the faulthandler module in issue11393.

--
nosy: +haypo, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26107] code.co_lnotab: use signed line number delta to support moving instructions in an optimizer

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26059] Integer Overflow in strop.replace()

[Ramin Farajpour Cami]

Ramin Farajpour Cami added the comment:

Not work ,i think should check sizeof of memory avoid overflow 
assert(new_len > 0 && new_len <= PY_SSIZE_T_MAX);

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25940] SSL tests failed due to expired svn.python.org SSL certificate

[koobs]

koobs added the comment:

Wow. *thank you* Georg and Martin :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Yury Selivanov]

New submission from Yury Selivanov:

This issue supersedes issue #6033.  I decided to open a new one, since the 
patch is about Python 3.6 (not 2.7) and is written from scratch.

The idea is to add new opcodes to avoid instantiation of BoundMethods.  The 
patch only affects method calls of Python functions with positional arguments.

I'm working on the attached patch in this repo: 
https://github.com/1st1/cpython/tree/call_meth2

If the patch gets accepted, I'll update it with the docs etc.


Performance Improvements


Method calls in micro-benchmarks are 1.2x faster:

### call_method ###
Avg: 0.330371 -> 0.281452: 1.17x faster

### call_method_slots ###
Avg: 0.333489 -> 0.280612: 1.19x faster

### call_method_unknown ###
Avg: 0.304949 -> 0.251623: 1.21x faster

Improvements in mini-benchmarks, such as Richards are less impressive, I'd say 
it's 3-7% improvement.  The full output of benchmarks/perf.py is here: 
https://gist.github.com/1st1/e00f11586329f68fd490

When the full benchmarks suite is run, some of them report that they were slow. 
 When I ran them separately several times, they all show no real slowdowns.  
It's just some of them (such as nbody) are highly unstable.

It's actually possible to improve the performance another 1-3% if we fuse 
__PyObject_GetMethod with ceval/LOAD_METHOD code.  I've tried this here: 
https://github.com/1st1/cpython/tree/call_meth4, however I don't like to have 
so many details of object.c into ceval.c.


Changes in the Core
---

Two new opcodes are added -- LOAD_METHOD and CALL_METHOD.  Whenever compiler 
sees a method call "obj.method(..)" with positional arguments it compiles it as 
follows:

  LOAD_FAST(obj)
  LOAD_METHOD(method)
  {call arguments}
  CALL_METHOD

LOAD_METHOD implementation in ceval looks up "method" on obj's type, and checks 
that it wasn't overridden in obj.__dict__.  Apparently, even with the __dict__ 
check this is still faster then creating a BoundMethod instance etc.  

If the method is found and not overridden, LOAD_METHOD pushes the resolved 
method object, and 'obj'.  If the method was overridden, the resolved method 
object and NULL are pushed to the stack.

CALL_METHOD then looks at the two stack values after call arguments.  If the 
first one isn't NULL, it means that we have a method call.


Why CALL_METHOD?


It's actually possible to hack CALL_FUNCTION to support LOAD_METHOD.  I've 
tried this approach in https://github.com/1st1/cpython/tree/call_meth3.  It 
looks like that adding extra checks in CALL_FUNCTION have negative impact on 
many benchmarks.  It's really easier to add another opcode.


Why only pure-Python methods?
-

LOAD_METHOD atm works only with methods defined in pure Python.  C methods, 
such as `list.append` are still wrapped into a descriptor, that creates a 
PyCFunction object on each attribute access.  I've tried to do that in 
https://github.com/1st1/cpython/tree/call_cmeth.  It does impact C method calls 
in a positive way, although my implementation is very hacky.  It still uses 
LOAD_METHOD and CALL_METHOD opcodes, so my idea is to consider merging this 
patch first, and then introduce the necessary refactoring of PyCFunction and 
MethodDesctiptors in a separate issue.


Why only calls with positional arguments?
-

As showed in "Why CALL_METHOD?", making CALL_FUNCTION to work with LOAD_METHOD 
slows it down.  For keyword and var-arg calls we have three more opcodes -- 
CALL_FUNCTION_VAR, CALL_FUNCTION_KW, and CALL_FUNCTION_VAR_KW.  I suspect that 
making them work with LOAD_METHOD would slow them down too, which will probably 
require us to add three (!) more opcodes for LOAD_METHOD.

And these kind of calls require much more overhead anyways, I don't expect them 
to be as optimizable as positional arg calls.

--
assignee: yselivanov
components: Interpreter Core
files: call_method_1.patch
keywords: patch
messages: 258204
nosy: benjamin.peterson, brett.cannon, gvanrossum, haypo, ncoghlan, yselivanov
priority: normal
severity: normal
stage: patch review
status: open
title: Speedup method calls 1.2x
type: performance
versions: Python 3.6
Added file: http://bugs.python.org/file41614/call_method_1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[STINNER Victor]

STINNER Victor added the comment:

> For keyword and var-arg calls we have three more opcodes -- 
> CALL_FUNCTION_VAR, CALL_FUNCTION_KW, and CALL_FUNCTION_VAR_KW.  I suspect 
> that making them work with LOAD_METHOD would slow them down too, which will 
> probably require us to add three (!) more opcodes for LOAD_METHOD.

I don't think that it's an issue to add 3 more opcodes for performance. If you 
prefer to limit the number of opcodes, you can pass a flag in arguments. For 
example, use 2 bytes for 2 arguments instead of only 1?

See also the (now old) WPython project which proposed kind of CISC instructions:
https://code.google.com/p/wpython/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Yury Selivanov]

Yury Selivanov added the comment:

> If you prefer to limit the number of opcodes, you can pass a flag in 
> arguments. For example, use 2 bytes for 2 arguments instead of only 1?

I tried two approaches:

1. Push one more leading NULL to the stack in LOAD_METHOD (which makes it push 
3 vals).  Then, in CALL_FUNCTION, detect when the callable is NULL and to the 
stuff that CALL_METHOD does.

2. Use a bitflag on oparg for CALL_FUNCTION and CALL_* opcodes.  This adds 
EXTENDED_ARG opcode overhead.

Long story short, the only option would be to add dedicated opcodes to work 
with LOAD_METHOD.  However, I'd prefer to first merge this patch, as it's 
relatively small and easy to review, and then focus on improving other things 
(keyword/*arg calls, C methods, etc).  This is just a first step.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26108] Calling PyInitialize with 2.7.11 on Windows x64 terminates process

[David Heffernan]

David Heffernan added the comment:

Thanks Eryk, everything you described happens exactly as you describe it. Much 
appreciated.

As it happens, I'm not distributing Python because I want to give my users the 
flexibility to use whatever version they please, and with whatever third party 
modules they want. So I'll just advise them to avoid 2.7.11.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6033] LOOKUP_METHOD and CALL_METHOD optimization

[Yury Selivanov]

Changes by Yury Selivanov :


--
superseder:  -> Speedup method calls 1.2x

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26103] Contradiction in definition of "data descriptor" between (dotted lookup behavior/datamodel documentation) and (inspect lib/descriptor how-to)

[SilentGhost]

Changes by SilentGhost :


--
assignee:  -> docs@python
components: +Documentation, Library (Lib)
nosy: +docs@python, yselivanov
stage:  -> needs patch
versions:  -Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[SilentGhost]

Changes by SilentGhost :


--
components: +Library (Lib)
nosy: +gvanrossum, pitrou
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25602] Add support for EVFILT_USER kqueue filter

[Jakub Klama]

Jakub Klama added the comment:

Hey, any news on this? I've created that issue over 2 months ago and it didn't 
get even commented since then.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[Antoine Pitrou]

Antoine Pitrou added the comment:

I'm frankly not sure that's a bug. If you want to filter out dotfiles, it is 
quite easy to do yourself. On the other hand, if pathlib always filtered them 
out, it would be more cumbersome to get a unified set of results with both 
dotfiles and non-dotfiles.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25957] sockaddr_l2 lacks CID, address type (AF_BLUETOOTH sockets)

[SilentGhost]

Changes by SilentGhost :


--
components: +Library (Lib)
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[Guido van Rossum]

Guido van Rossum added the comment:

I've thought about this too, and I've decided it's a feature. As Antoine says, 
it's easy enough to filter the dot files out if you don't want them; it's 
slightly complicated to include them if the default is to skip them.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25602] Add support for EVFILT_USER kqueue filter

[SilentGhost]

Changes by SilentGhost :


--
nosy: +haypo
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26038] zipfile cannot handle zip files where the archive size for a file does not match actual contents

[SilentGhost]

Changes by SilentGhost :


--
nosy: +alanmcintyre, serhiy.storchaka, twouters
versions: +Python 3.5, Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Guido van Rossum]

Guido van Rossum added the comment:

I like this idea! I like the limitations to positional-only calls. I do think 
that it would be nice if we could speed up C calls too -- today, 
s.startswith('abc') is slower than s[:3] == 'abc' precisely because of the 
lookup. But I'm all for doing this one step at a time, so we can be sure it is 
solid before taking the next step(s).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25491] ftplib.sendcmd only accepts string

[SilentGhost]

Changes by SilentGhost :


--
nosy: +giampaolo.rodola
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Brett Cannon]

Changes by Brett Cannon :


--
nosy: +twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Remy Roy]

New submission from Remy Roy:

On Windows, os.scandir will keep a handle on the directory being scanned until 
the iterator is exhausted. This behavior can cause various problems if try to 
use some filesystem calls like os.chmod or os.remove on the directory while the 
handle is still being kept.

There are some use cases where the iterator is not going to be exhausted like 
looking for a specific entry in a directory and breaking from the loop 
prematurely.

This behavior should at least be documented.  Alternatively, it might be 
interesting to provide a way prematurely end the scan without having to exhaust 
it and close the handle.

As a workaround, you can force the exhaustion after you are done with the 
iterator with something like:

for entry in iterator:
pass

This is going to affect os.walk as well since it uses os.scandir .

The original github issue can be found on 
https://github.com/benhoyt/scandir/issues/58 .

--
components: Windows
messages: 258212
nosy: paul.moore, remyroy, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: On Windows, os.scandir will keep a handle on the directory until the 
iterator is exhausted
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Brett Cannon]

Changes by Brett Cannon :


--
nosy: +dino.viehland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24945] Expose Py_TPFLAGS_ values from Python

[SilentGhost]

Changes by SilentGhost :


--
nosy: +benjamin.peterson
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Ben Hoyt]

Changes by Ben Hoyt :


--
nosy: +benhoyt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25753] Reference leaks in test_smtplib

[SilentGhost]

SilentGhost added the comment:

Serhiy, I cannot reproduce these leaks on recent checkout. Is this something 
that's still happens for you?

--
nosy: +SilentGhost

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26112] Error on example using "dialect" parameter. have to be: "dialect=dialect"

[Василь Коломієць]

New submission from Василь Коломієць:

with open('example.csv') as csvfile:
dialect = csv.Sniffer().sniff(csvfile.read(1024))
csvfile.seek(0)
reader = csv.reader(csvfile, dialect)
# ... process CSV file contents here ...
--

have to be:
...
reader = csv.reader(csvfile, dialect=dialect)
 # ... process CSV file contents here ...

It's here:
https://docs.python.org/3.4/library/csv.html

--
assignee: docs@python
components: Documentation
messages: 258214
nosy: docs@python, Василь Коломієць
priority: normal
severity: normal
status: open
title: Error on example using "dialect" parameter. have to be: "dialect=dialect"
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26112] Error on example using "dialect" parameter. have to be: "dialect=dialect"

[SilentGhost]

SilentGhost added the comment:

No, the shown code works just fine.

--
nosy: +SilentGhost
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25668] Deadlock in logging caused by a possible race condition with "format"

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26113] pathlib p.match('') should return False rather than raising exception

[JitterMan]

New submission from JitterMan:

One can use '*' as an 'accept all' pattern to match(). It would be nice to also 
use '' as a 'reject all' pattern. These 'accept all' and 'reject all' rules are 
useful as defaults. Currently passing '' to match() causes an exception. While 
it is easy enough to catch the exception, the need to constrains the way 
match() must be called and makes the calling code needlessly complicated and 
fussy.

--
files: example.py
messages: 258216
nosy: jitterman
priority: normal
severity: normal
status: open
title: pathlib p.match('') should return False rather than raising exception
type: enhancement
versions: Python 3.5
Added file: http://bugs.python.org/file41615/example.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26113] pathlib p.match('') should return False rather than raising exception

[SilentGhost]

Changes by SilentGhost :


--
components: +Library (Lib)
nosy: +gvanrossum, pitrou
versions: +Python 3.6 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26058] PEP 509: Add ma_version to PyDictObject

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26113] pathlib p.match('') should return False rather than raising exception

[Guido van Rossum]

Guido van Rossum added the comment:

What's the use case for a "reject all" pattern? Why do you want to call glob 
with an argument that causes it do do a bunch of pointless work but then return 
an empty result set?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26114] Rewrite math.erf() and math.erfc() from scratch

[Brett Cannon]

New submission from Brett Cannon:

If you look Modules/mathmodule.c you will notice there is a comment that goes 
with erf() and erfc() stating that the algorithms were taken from a book 
entitled 'Numerical Recipes'. Unfortunately that book has a license dictating 
that any information from the book is only allowed for non-commercial use; 
commercial use requires negotiating a license 
(http://numerical.recipes/aboutNR3license.html). That's bad for anyone who has 
a commercial distribution of Python as that's a special requirement they have 
to follow.

It would be best to do a clean room implementation of both math.erf() and 
math.erfc() that does not use information from 'Numerical Recipes' in order to 
not be violating that license. That way Python can be sold commercially without 
having to negotiate a separate license just for those two functions.

Unfortunately this code exists since at least Python 2.7, so I have flagged all 
Python releases as needing the eventual clean room implementation applied to it 
(although Python 3.2 goes out of security maintenance next month so I don't 
know how critical it is to fix that far back).

--
components: Library (Lib)
messages: 258218
nosy: benjamin.peterson, brett.cannon, eric.smith, georg.brandl, larry, 
lemburg, mark.dickinson, stutzbach
priority: release blocker
severity: normal
stage: needs patch
status: open
title: Rewrite math.erf() and math.erfc() from scratch
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26114] Rewrite math.erf() and math.erfc() from scratch

[Brett Cannon]

Changes by Brett Cannon :


--
nosy: +peter, teoliphant

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26114] Rewrite math.erf() and math.erfc() from scratch

[Brett Cannon]

Changes by Brett Cannon :


--
nosy:  -peter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Eryk Sun]

Eryk Sun added the comment:

If you own the only reference you can also delete the reference, which 
deallocates the iterator and closes the handle.

Can you provide concrete examples where os.remove and os.chmod fail? At least 
in Windows 7 and 10 the directory handle is opened with the normal read and 
write sharing, but also with delete sharing. This sharing mode is fairly close 
to POSIX behavior (an important distinction is noted below). I get the 
following results in Windows 10:

>>> import os, stat
>>> os.mkdir('test')
>>> f = open('test/file1', 'w'); f.close()
>>> f = open('test/file2', 'w'); f.close()
>>> it = os.scandir('test')
>>> next(it)


rename, chmod, and rmdir operations succeed:

>>> os.rename('test', 'spam')
>>> os.chmod('spam', stat.S_IREAD)
>>> os.chmod('spam', stat.S_IWRITE)
>>> os.remove('spam/file1')
>>> os.remove('spam/file2')
>>> os.rmdir('spam')

Apparently cached entries can be an issue, but this caching is up to WinAPI 
FindNextFile and the system call NtQueryDirectoryFile:

>>> next(it)


An important distinction is that a deleted file in Windows doesn't actually get 
unlinked until all handles and kernel pointer references are closed. Also, once 
the delete disposition is set, no *new* handles can be created for the existing 
file or directory (all access is denied), and a new file or directory with same 
name cannot be created.

>>> os.listdir('spam')
Traceback (most recent call last):
  File "", line 1, in 
PermissionError: [WinError 5] Access is denied: 'spam'

>>> f = open('spam', 'w')
Traceback (most recent call last):
  File "", line 1, in 
PermissionError: [Errno 13] Permission denied: 'spam'

If we had another handle we could use that to rename "spam" to get it out of 
the way, at least. Without that, AFAIK, all we can do is deallocate the 
iterator or wait for it to be exhausted, which closes the handle and thus 
allows Windows to finally unlink "spam":

>>> next(it)
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

Creating a new file named "spam" is allowed now:

>>> f = open('spam', 'w')
>>> f.close()

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26114] Rewrite math.erf() and math.erfc() from scratch

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[Yury Selivanov]

Yury Selivanov added the comment:

> I like this idea! I like the limitations to positional-only calls. I do think 
> that it would be nice if we could speed up C calls too -- today, 
> s.startswith('abc') is slower than s[:3] == 'abc' precisely because of the 
> lookup. But I'm all for doing this one step at a time, so we can be sure it 
> is solid before taking the next step(s).

Yes, I think we can make `.method(..)` calls much faster with 
LOAD_METHOD.  Since those types lack `__dict__` and their `__class__` is 
read-only, we can use a far better optimized code path without extra lookups 
and creation of BoundMethod/PyCFunction objects.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[JitterMan]

JitterMan added the comment:

The same issues applies with '**', where it is harder to work around. The shell 
version of '**' prunes out hidden files and directories at all levels, whereas 
the pathlib.glob() includes all hidden directories.

Major surgery would be required to filter out hidden files and directories from 
the output of pathlib.glob(), which would be expensive and complicated. Perhaps 
an option is called for that would allow pathlib.glob() to prune its search to 
only non-hidden directories if requested. And if that seems like a good idea, 
perhaps there could also be options that specify that only files or only 
directories should be returned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26110] Speedup method calls 1.2x

[STINNER Victor]

STINNER Victor added the comment:

I'm happy to see people working on optimizing CPython ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26115] pathlib.glob('**') returns only directories

[JitterMan]

New submission from JitterMan:

The title says it all.

The shell version of '*' and '**' return both directories and files.
Path('.').glob('*') returns both directories and files, but 
Path('.').glob('**') returns only directories. That seems wrong to me.

--
messages: 258223
nosy: jitterman
priority: normal
severity: normal
status: open
title: pathlib.glob('**') returns only directories
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25753] Reference leaks in test_smtplib

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The result of searching leaks is not stable. I ran the test 10 times and got 
failures 3 times.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Martin Panter]

Martin Panter added the comment:

Remy, is this the same problem described in Issue 25994? There a close() method 
(like on generators) and/or context manager support is proposed for the 
scandir() iterator. Perhaps we can keep this issue open for adding a warning to 
the documentation, and the other issue can be for improving the API in 3.6.

--
assignee:  -> docs@python
components: +Documentation
dependencies: +File descriptor leaks in os.scandir()
nosy: +docs@python, martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Remy Roy]

Remy Roy added the comment:

I believe Eryk's explanation on how a file in Windows doesn't actually get 
unlinked until all handles and kernel pointer references are closed is spot on 
about the problem I had.

I had a complex example that could probably have been simplified to what Eryk 
posted.

That behavior on Windows is quite counterintuitive. I'm not sure about what can 
be done to help it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Василь Коломієць]

New submission from Василь Коломієць:

with open('example.csv') as csvfile:
dialect = csv.Sniffer().sniff(csvfile.read(1024))
csvfile.seek(0)
reader = csv.reader(csvfile, dialect)
# ... process CSV file contents here ...
--

have to be:
...
reader = csv.reader(csvfile, dialect=dialect)
 # ... process CSV file contents here ...

It's here:
https://docs.python.org/3.4/library/csv.html

--
assignee: docs@python
components: Documentation
messages: 258227
nosy: Vasyl Kolomiets, docs@python
priority: normal
severity: normal
status: open
title: CSV-module. The example code don't work.Have to be: reader = 
csv.reader(csvfile, dialect=dialect)
type: enhancement
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Remy Roy]

Remy Roy added the comment:

This issue is not same as Issue 25994 but it is quite related. Some kind of 
close() method and/or context manager support could help here as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Василь Коломієць]

Василь Коломієць added the comment:

Traceback (most recent call last):
  File "F:\Program Files (x86)\Pyton3_4\=Py\NovaPoshta\View#1-check-Trojan.py", 
line 11, in 
for row in readed:
  File "F:\Program Files (x86)\Pyton3_4\lib\csv.py", line 118, in __next__
d = dict(zip(self.fieldnames, row))
TypeError: zip argument #1 must support iteration

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18018] SystemError: Parent module '' not loaded, cannot perform relative import

[Chris Angelico]

Chris Angelico added the comment:

This still happens in current CPython, and doesn't even require a package:

rosuav@sikorsky:~$ echo 'from . import x' >syserr.py
rosuav@sikorsky:~$ python3 syserr.py
Traceback (most recent call last):
  File "syserr.py", line 1, in 
from . import x
SystemError: Parent module '' not loaded, cannot perform relative import


This just caused some confusion for a student of mine who had migrated from 
Python 2. Converting intra-package imports from "import spam" to "from . import 
spam" made them work; converting a non-package import the same way caused this 
problem. ImportError would be massively preferable.

But I'm not understanding something here. If I set __package__ inside the 
module, the behaviour changes accordingly:

rosuav@sikorsky:~$ cat syserr.py 
__package__ = 'sys'
from . import version
print(version)
rosuav@sikorsky:~$ python3 syserr.py 
3.6.0a0 (default:ac94418299bd+, Jan 15 2016, 08:44:02) 
[GCC 4.9.2]


Leaving __package__ unset has it implicitly be None. Whether it's None or '', 
the package checks should simply not be made - see three lines above the line 
Brett linked to (in today's code, that's 
https://hg.python.org/cpython/file/ac94418299b/Lib/importlib/_bootstrap.py#l925 
- hasn't changed). So I'm not sure why the checks are even happening. However, 
that probably means more about my exploration and testing than it does about 
the code; editing the text of the message doesn't result in a change, so I'm 
obviously not updating the frozen version.

--
nosy: +Rosuav
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24752] SystemError when importing from a non-package directory

[Chris Angelico]

Chris Angelico added the comment:

Linking this issue with #18018 - same problem, same situation.

--
nosy: +Rosuav

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18018] SystemError: Parent module '' not loaded, cannot perform relative import

[Wil D'Amato]

Changes by Wil D'Amato :


--
nosy: +auslander1970

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24752] SystemError when importing from a non-package directory

[Brett Cannon]

Changes by Brett Cannon :


--
superseder:  -> SystemError: Parent module '' not loaded, cannot perform 
relative import

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24752] SystemError when importing from a non-package directory

[Brett Cannon]

Changes by Brett Cannon :


--
resolution:  -> duplicate
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26117] Close directory descriptor in scandir iterator on error

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Since scandir iterator has no close method, the most safe way to avoid file 
descriptors leaks when use os.scandir() is accumulating yielded results into a 
list before processing them:

entries = list(os.scandir(path))

But this doesn't save us from all leaks. If an error happened during yielding 
next entry, opened file descriptor left open.

Proposed patch makes scandir to close the file descriptor not only when an 
iteration is exhausted, but when any error (expected OSError or MemoryError) is 
raised.

This is needed to fix possible leaks in os.walk() in 3.5 (see also issue25995).

--
components: Extension Modules
files: scandir_closedir_on_error.patch
keywords: patch
messages: 258232
nosy: benhoyt, gvanrossum, haypo, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Close directory descriptor in scandir iterator on error
type: resource usage
versions: Python 3.5, Python 3.6
Added file: http://bugs.python.org/file41616/scandir_closedir_on_error.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18018] SystemError: Parent module '' not loaded, cannot perform relative import

[Brett Cannon]

Changes by Brett Cannon :


--
assignee: flox -> brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25994] File descriptor leaks in os.scandir()

[Martin Panter]

Martin Panter added the comment:

I definitely support adding a close() method, and a ResourceWarning if the 
iterator is not exhausted when it is deallocated. Adding context manager 
support is probably worthwhile as well, though there is one disadvantage: it 
would be hard to implement scandir() as a plain generator, because generators 
don’t support the context manager protocol. But C Python’s implementation is in 
C, and even in native Python you could implement it as a custom iterator class.

I suggest using this issue for adding the new API(s), presumably to 3.6 only, 
and Issue 26111 for adding a warning to the existing 3.5 documentation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[SilentGhost]

SilentGhost added the comment:

Vasyl, this line in your code is causing the problem:

for row in readed:

The error reasonably clear states that it's due to your `fieldnames' parameter 
being non-iterable. Again, this has nothing whatsoever to do with `dialect' 
argument and documentation of Sniffer. 

Please, do not open new issues about this. This is not a issue with csv library 
or its documentation. Try to use use stackoverflow or python-tutor list to get 
help on this.
https://mail.python.org/pipermail/tutor/

--
nosy: +SilentGhost
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Martin Panter]

Martin Panter added the comment:

Can you explain how it is different? The way I see it, both problems are about 
the scandir() iterator holding an open reference (file descriptor or handle) to 
a directory/folder, when the iterator was not exhausted, but the caller no 
longer needs it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Eryk Sun]

Eryk Sun added the comment:

> That behavior on Windows is quite counterintuitive.

It's counter-intuitive from a POSIX point of view, in which anonymous files are 
allowed. In contrast, Windows allows any existing reference to unset the delete 
disposition, so the name cannot be unlinked until all references are closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26115] pathlib.glob('**') returns only directories

[SilentGhost]

SilentGhost added the comment:

It is, however, exactly what documentation says it should do:

> The “**” pattern means “this directory and all subdirectories, recursively”.

--
nosy: +SilentGhost
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25995] os.walk() consumes a lot of file descriptors

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, OSError, MemoryError and KeyboardInterrupt can be raised during iterating 
scandir object or calling is_dir() or is_symlink() methods of the result. They 
stop iterating and left file descriptor open. If close file descriptor on error 
during iteration (issue26117), patch 1 becomes more safe, because it minimizes 
the lifetime of opened descriptor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26102] access violation in PyErrFetch if tcur==null in PyGILState_Release

[SilentGhost]

Changes by SilentGhost :


--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25994] File descriptor leaks in os.scandir()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If scandir() is implemented as native Python generator (see for example 
issue25911), it could easily be converted to context manager:

def scandir(path):
return contextlib.closing(native_scandir(path))

def native_scandir(path):
...
yield ...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Василь Коломієць]

Василь Коломієць added the comment:

Thanx a lot.
And sorry.
But with ..., dialect=dialect,  the same code is working.
.
Ok. i'll try to understand.
With best wishes - VK.

С уважением,
Василий Коломиец
+38050 45 22 559

2016-01-15 0:18 GMT+02:00 SilentGhost :

>
> SilentGhost added the comment:
>
> Vasyl, this line in your code is causing the problem:
>
> for row in readed:
>
> The error reasonably clear states that it's due to your `fieldnames'
> parameter being non-iterable. Again, this has nothing whatsoever to do with
> `dialect' argument and documentation of Sniffer.
>
> Please, do not open new issues about this. This is not a issue with csv
> library or its documentation. Try to use use stackoverflow or python-tutor
> list to get help on this.
> https://mail.python.org/pipermail/tutor/
>
> --
> nosy: +SilentGhost
> resolution:  -> not a bug
> stage:  -> resolved
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___
>

--
title: CSV-module. The example code don't work.Have to be: reader = 
csv.reader(csvfile, dialect=dialect) -> CSV-module. The example code don't 
work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Georg Brandl]

Georg Brandl added the comment:

We'll need at least the two full examples (one not working, one working) to 
make a call here.

--
nosy: +georg.brandl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25994] File descriptor leaks in os.scandir()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Since the close() method can be added only in 3.6, but we have leaks in 3.5 
too, I suggest to closed file descriptor if error happened during iteration 
(issue26117). This will allow to write safe code even in 3.5.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26118] String performance issue using single quotes

[poostenr]

New submission from poostenr:

There appears to be a significant performance issue between the following two 
statements. Unable to explain performance impact.

s = "{0},".format(columnvalue)   # fast
s = "'{0}',".format(columnvalue) # ~30x slower

So far, no luck trying to find other statements to improve performance, such as:
s = "\'{0}\',".format(columnvalue)
s = "'" + "%s" %(columnvalue) + "'"+","
s = "{0}{1}{2},".format("'",columnvalue,"'")

--
components: Windows
messages: 258243
nosy: paul.moore, poostenr, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: String performance issue using single quotes
type: performance
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24950] FAIL: test_expanduser when $HOME=/

[SilentGhost]

SilentGhost added the comment:

For posixpath the fix is straightforward: just skip that assert if the home 
ends up being '/' (the '/' is tested above).

For pathlib, I'm not entirely sure what the fix should be.

--
nosy: +SilentGhost, pitrou
versions: +Python 3.6 -Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25994] File descriptor leaks in os.scandir()

[Martin Panter]

Martin Panter added the comment:

Contextlib.closing() cannot be used in general, because it doesn’t inherit the 
iterator protocol from the wrapped generator. So I think you really need a 
class that implements __exit__(), __iter__(), etc at the same time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[Guido van Rossum]

Guido van Rossum added the comment:

If you have that use case you're probably better of using os.walk() so you are 
not limited to a prune strategy that can be expressed using a single glob 
pattern (e.g. maybe I want to ignore .git, .hg and __pycache__ but descend into 
everything else).

I think we would also welcome adding a walk() method to pathlib. But let's 
please leave glob() alone.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26118] String performance issue using single quotes

[ubehera]

Changes by ubehera :


--
nosy: +ubehera

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26118] String performance issue using single quotes

[Eric V. Smith]

Eric V. Smith added the comment:

Please show us how you're measuring the performance. Also, please show the 
output of "python -V".

--
components: +Interpreter Core -Windows
nosy: +eric.smith -paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25682] __package__ not set to None under pdb in Python 3

[SilentGhost]

Changes by SilentGhost :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26111] On Windows, os.scandir will keep a handle on the directory until the iterator is exhausted

[Remy Roy]

Remy Roy added the comment:

>From my point of view, Issue 25994 is about the potential file 
>descriptor/handle leaks and this issue is about being unable to perform some 
>filesystem calls because of a hidden unclosed file descriptor/handle.

I am not going to protest if you want to treat them as the same issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Василь Коломієць]

Василь Коломієць added the comment:

I am very happy!
Thx for attention to my opinion.
Sorry for my english.

so - in attachment two codes and source csv-file.

With best wishes,
Vasyl Kolomiets.
+38050 45 22 559

2016-01-15 0:38 GMT+02:00 Georg Brandl :

>
> Georg Brandl added the comment:
>
> We'll need at least the two full examples (one not working, one working)
> to make a call here.
>
> --
> nosy: +georg.brandl
>
> ___
> Python tracker 
> 
> ___
>

--
Added file: http://bugs.python.org/file41617/may_be-error_csv_exmpl.py
Added file: http://bugs.python.org/file41618/from_to_proportions.csv
Added file: http://bugs.python.org/file41619/may_be-good_csv_exmpl.py

___
Python tracker 

___from tkinter import *
from PIL import Image
import csv

filename = "from_to_proportions.csv"
with open(filename) as csvfile:
dialect=csv.Sniffer().sniff(csvfile.read(1024))
csvfile.seek(0)
print(dialect)
readed = csv.DictReader(csvfile, dialect)
for row in readed:
print(row)
print (row["from_id"],row["from_name"],row['to_id'],row['to_name'],
   row['barcodes_percentage'],row['rows_to_be'])

from_id;from_name;to_id;to_name;barcodes_percentage;rows_to_be
1;Київ;2;Дніпро;75%;4
1;Київ;3;Запоріжжя;10%;4
1;Київ;4;Одеса;15%;4
1;Київ;5;Львів;0%;4
2;Дніпро;1;Київ;50%;4
2;Дніпро;3;Запоріжжя;10%;4
2;Дніпро;4;Одеса;25%;4
2;Дніпро;5;Львів;15%;4
3;Запоріжжя;1;Київ;0%;4
3;Запоріжжя;2;Дніпро;33%;4
3;Запоріжжя;4;Одеса;33%;4
3;Запоріжжя;5;Львів;34%;4
4;Одеса;1;Київ;25%;4
4;Одеса;2;Дніпро;25%;4
4;Одеса;3;Запоріжжя;25%;4
4;Одеса;5;Львів;25%;4
5;Львів;1;Київ;0%;4
5;Львів;2;Дніпро;50%;4
5;Львів;3;Запоріжжя;15%;4
5;Львів;4;Одеса;35%;4
from tkinter import *
from PIL import Image
import csv

filename = "from_to_proportions.csv"
with open(filename) as csvfile:
dialect=csv.Sniffer().sniff(csvfile.read(1024))
csvfile.seek(0)
print(dialect)
readed = csv.DictReader(csvfile, dialect=dialect)
for row in readed:
print(row)
print (row["from_id"],row["from_name"],row['to_id'],row['to_name'],
   row['barcodes_percentage'],row['rows_to_be'])

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26116] CSV-module. The example code don't work. Have to be: reader = csv.reader(csvfile, dialect=dialect)

[Василь Коломієць]

Changes by Василь Коломієць :


Added file: http://bugs.python.org/file41620/CSV-ex.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26118] String performance issue using single quotes

[poostenr]

poostenr added the comment:

My initial observations with my Python script using:

s = "{0},".format(columnvalue)   # fast
Processed ~360MB of data from 2:16PM - 2:51PM (35 minutes, ~10MB/min)
One particular file 6.5MB took ~1 minute.

When I changed this line of code to:
s = "'{0}',".format(columnvalue) # ~30x slower (1 min. vs 30 min.)
Same particular file of 6.5MB took ~30 minutes (228KB/min).

My Python environment is:
C:\Data>python -V
Python 3.5.1 :: Anaconda 2.4.1 (32-bit)

I did some more testing with a very simplified piece of code, but is not 
conclusive. But there is a significant jump when I introduce the single quotes 
where I see a jump from 0m2.410s to 0m3.875s.

$ python -V
Python 3.5.1

// 
// s='test'
// for x in range(1000):
// y = "{0}".format(s)

$ time python test.py

real0m2.410s
user0m2.356s
sys 0m0.048s

// s='test'
// for x in range(1000):
// y = "'%s'" % (s)

$ time python test2.py 

real0m2.510s
user0m2.453s
sys 0m0.051s

// s='test'
// for x in range(1000):
// y = "'{0}'".format(s)

$ time python test3.py 

real0m3.875s
user0m3.819s
sys 0m0.048s

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25940] SSL tests failed due to expired svn.python.org SSL certificate

[Martin Panter]

Martin Panter added the comment:

In the 3.3 branch, I got a failure in test_ssl.ThreadedTests.test_dh_params(): 
“SSLError: [SSL] dh key too small (_ssl.c:548)”. But the failure also happens 
in the revision before my merge, so I think it must be a separate problem.

Also, I am seeing Windows 7+ buildbots are failing the new version of 
test_connect_ex_error(). On Linux, connecting to self-signed.pythontest.net:444 
gives EHOSTUNREACH, but it seems we get ETIMEDOUT on Windows:

AssertionError: 10060 not found in (10061, 10065, 10035)

I will update the test to check for that error code as well, since all it 
really needs to test is that _some_ sensible error code is returned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23606] ctypes.util.find_library("c") no longer makes sense

[Antoine Pitrou]

Antoine Pitrou added the comment:

Does it mean `cdll.msvcrt` is not the standard way to access the C symbols 
anymore? Could you update the docs to reflect the current guidelines?
(https://docs.python.org/3/library/ctypes.html)

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23606] ctypes.util.find_library("c") no longer makes sense

[Steve Dower]

Steve Dower added the comment:

Strictly there's nothing incorrect about the docs, and `cdll.msvcrt` is no more 
incorrect than it has been since Python 2.4 or so (whenever we  stopped using 
VC6). It will find the DLL and you can call the functions, but it isn't the 
same DLL as the exports in the `msvcrt` module will call.

Might be worth a note pointing this out, but it's probably better off with a 
different example. I'll try and think of something, but I'm fairly distracted 
at the moment and would appreciate suggestions. Otherwise I'll get there 
eventually.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26118] String performance issue using single quotes

[Eric V. Smith]

Eric V. Smith added the comment:

I see a small difference, but I think it's related to the fact that in the 
first example you're concatenating 2 strings (',' and the result of {0}), and 
in the 2nd example it's 3 strings ("'", {0}, "',"):

$ echo '",{0}".format(x)'
",{0}".format(x)
$ python -m timeit -s 'x=4' '",{0}".format(x)'
100 loops, best of 3: 0.182 usec per loop

$ echo '"'\''{0}'\'',".format(x)'
"'{0}',".format(x)
$ python -m timeit -s 'x=4' '"'\''{0}'\'',".format(x)'
100 loops, best of 3: 0.205 usec per loop

If you see a factor of 30x difference in your code, I suspect it's not related 
to str.format(), but some other processing in your code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26113] pathlib p.match('') should return False rather than raising exception

[JitterMan]

JitterMan added the comment:

I don't know that passing '' as a pattern to glob() makes much sense, but it is 
useful when passed to match(). Doing so allows me to build a filter that can 
easily and naturally be disabled. For example, the following allows me to get 
all the items in a directory that both match a select pattern and do not match 
a reject pattern:

def ls(dir, select='*', reject='.*'):
return (p for p in dir.glob(select) if not p.match(reject))

By default this function does not return hidden files or directories. It would 
seem like this restriction could be removed by passing reject='', but that 
generates an exception. Working around the exception makes the code noticeably 
more complicated.

Perhaps the question should really be 'what is the benefit of raising an 
exception when an empty glob string is encountered?'. I cannot think of any.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com