[Python-Dev] Should we allow IPv6 scope in urllib: http://[IPv6%scope]/path?

2019-10-21 Thread Victor Stinner 
Hi,

I wrote a PR to fix the following urllib security vulnerability:

"urlparse of urllib returns wrong hostname"
https://bugs.python.org/issue36338
https://github.com/python/cpython/pull/16780

While writing my fix, I found another issue about "[" and "]"
characters in the user:password section of an URL:

"urllib IPv6 parsing fails with special characters in passwords"
https://bugs.python.org/issue33342

My PR tries to validate the "scope" part of
"http://[IPv6%scope]/...": reject "%", "[" and "]" in scope. But I'm
not sure that Python should really support the scope in an URL. Should
we just reject URL with "%scope"? Or if we allow it, which characters
should be allowed and/or rejected?

It seems like Firefox and Chromium don't support an IPv6 with as a
scope: when I type http://[::1%1]/ : they open a Google search on this
URL.

I tested Python urllib.request.urlopen() with my PR:
http://[::1%1]:8080/ works as expected: it opens a connection to the
IPv6 localhost in the loopback interface (TCP port 8080).

Currently, my PR allows "%scope" but it rejects "%", "[" and "]"
characters in the scope.

I let you go through these 2 RFC about IPv6 scope / "zone identifier":

https://tools.ietf.org/html/rfc4007
https://tools.ietf.org/html/rfc6874

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T7NJBO6B32KZZ5KAXJXHT7CTKU7634UU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Need help to fix test_asyncio issues

2019-10-21 Thread Victor Stinner 
Hi,

Right now, there are 14 open issues with "test_asyncio" in the title.
Many test_asyncio tests have race conditions. I'm trying to fix them
one by one, but it takes time, and then new tests are added with new
race condition :-( For example, the following new test is failing
randomly on Windows:

"Windows: test_asyncio: test_huge_content_recvinto() fails randomly
with ProactorEventLoop" is failing randomly since 6 months:
https://bugs.python.org/issue36732

test_asyncio uses more and more functional tests which is a good
thing. In the early days of asyncio, most tests mocked more than half
of asyncio to really be "unit test". But at the end, the test tested
more mocks than asyncio... The problem of functional tests is that
it's hard to design them properly to avoid all race conditions,
especially when you consider multiplatform (Windows, macOS, Linux,
FreeBSD, etc.).

It would help me if someone could try to investigate these issues,
provide a reliable way to reproduce them, and propose a fix. (Simply
saying that you can reproduce the test and that you would like to work
on an issue doesn't really help, sorry.)

Recently, I started to experiment "./python -m test [options] -F
-j100" to attempt to reproduce some tricky race conditions: -j100
spawns 100 worker processes in parallel and -F stands for --forever
(run tests in loop and stop at the first failure). I was surprised
that my Fedora 30 didn't burn in blame. In fact, the GNOME desktop
remains responsible even with a system load higher than 100. The Linux
kernel (5.2) is impressive! Under such high system load (my laptop has
8 logical CPUs), race conditions are way more likely.

The problem of test_asyncio is that it's made of 2160 tests, see:

   ./python -m test test_asyncio --list-cases

You may want to only run a single test case (class) or even a single
test method: see --match option which can be used multiple times to
only run selected test classes or selected test methods. See also
--matchfile which is similar but uses a file. Example:

$ ./python -m test test_asyncio --list-cases > cases
# edit cases
$ ./python -m test test_asyncio --matchfile=cases

test_asyncio is one of the most unstable test: I'm getting more and
more buildbot-status emails about test_asyncio... likely because we
fixed most of the other race conditions which is a good thing ;-)

Some issues look to be specific to Windows, but it should be possible
to reproduce most issues on Linux as Linux. Sometimes, it's just that
some specific Windows buildbot workers are slower than other buildbot
workers.

Good luck ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGEOKWD3PBWIL2LPZWZ6MMRANN5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Getting function arguments of PyCFunctionObject

2019-10-21 Thread Petr Viktorin 

On 2019-10-19 00:31, Sümer Cip wrote:

Hi everyone,

I have been working on some feature of a deterministic 
profiler(github.com/sumerc/yappi ). The 
feature is about getting arguments for a given set of function names. 
For example: you can define something like foo 1,3,arg_name and when foo 
function is called, profiler will simply collect the first, third from 
*args and a named argument arg_name from *kwargs.


For Python functions I am using following approach: Please note that the 
code below is executing in the PyTrace_CALL event of the profiler in C side:
Look co_argcount and co_varnames to determine the names of arguments and 
then use these names to retrieve values from f_locals. It seems to be 
working fine for now. My first question is: Is there a better way to do 
this?


And for C functions, I am totally in dark. I have played with 
f_valuestack and retrieve some values from there but the indexes seem to 
change from Python version to version and also I think there is no way 
of getting named arguments...


I have been dealing with this for a while, so there might be unclear 
points in my explanation. I would really appreciate if anyone can help 
me on a correct direction on this.




Hi,
The best way we know of getting a callable's arguments is 
inspect.signature: 
https://docs.python.org/3.5/library/inspect.html#inspect.signature


Of course, some callables can lie about their arguments, some might have 
information missing, and sometimes there even are good reasons to do 
such things.
If you can't use inspect.signature directly, your best bet is probably 
to study its implementation. It's complex but hopefully well-enough 
commented. Just be aware it can change in the future, as we find better 
ways of documenting and finding signatures. And if you find a possible 
improvement, send it here!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UR6S46WVGAVXSYKG5736VC24LJ2BHSID/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Need help to fix test_asyncio issues

2019-10-21 Thread Kyle Stanley 
> Recently, I started to experiment "./python -m test [options] -F
-j100" to attempt to reproduce some tricky race conditions: -j100
spawns 100 worker processes in parallel and -F stands for --forever
(run tests in loop and stop at the first failure).

Interesting, even as someone who has recently worked within
asyncio a decent amount recently, I was entirely unaware of
this method of producing race conditions in the tests. If continued
experimentation shows this to be a reliable means of reproducing
various race conditions, it might be worth adding this to
https://devguide.python.org/runtests/.

Thanks for sharing!


On Mon, Oct 21, 2019 at 7:08 AM Victor Stinner  wrote:

> Hi,
>
> Right now, there are 14 open issues with "test_asyncio" in the title.
> Many test_asyncio tests have race conditions. I'm trying to fix them
> one by one, but it takes time, and then new tests are added with new
> race condition :-( For example, the following new test is failing
> randomly on Windows:
>
> "Windows: test_asyncio: test_huge_content_recvinto() fails randomly
> with ProactorEventLoop" is failing randomly since 6 months:
> https://bugs.python.org/issue36732
>
> test_asyncio uses more and more functional tests which is a good
> thing. In the early days of asyncio, most tests mocked more than half
> of asyncio to really be "unit test". But at the end, the test tested
> more mocks than asyncio... The problem of functional tests is that
> it's hard to design them properly to avoid all race conditions,
> especially when you consider multiplatform (Windows, macOS, Linux,
> FreeBSD, etc.).
>
> It would help me if someone could try to investigate these issues,
> provide a reliable way to reproduce them, and propose a fix. (Simply
> saying that you can reproduce the test and that you would like to work
> on an issue doesn't really help, sorry.)
>
> Recently, I started to experiment "./python -m test [options] -F
> -j100" to attempt to reproduce some tricky race conditions: -j100
> spawns 100 worker processes in parallel and -F stands for --forever
> (run tests in loop and stop at the first failure). I was surprised
> that my Fedora 30 didn't burn in blame. In fact, the GNOME desktop
> remains responsible even with a system load higher than 100. The Linux
> kernel (5.2) is impressive! Under such high system load (my laptop has
> 8 logical CPUs), race conditions are way more likely.
>
> The problem of test_asyncio is that it's made of 2160 tests, see:
>
>./python -m test test_asyncio --list-cases
>
> You may want to only run a single test case (class) or even a single
> test method: see --match option which can be used multiple times to
> only run selected test classes or selected test methods. See also
> --matchfile which is similar but uses a file. Example:
>
> $ ./python -m test test_asyncio --list-cases > cases
> # edit cases
> $ ./python -m test test_asyncio --matchfile=cases
>
> test_asyncio is one of the most unstable test: I'm getting more and
> more buildbot-status emails about test_asyncio... likely because we
> fixed most of the other race conditions which is a good thing ;-)
>
> Some issues look to be specific to Windows, but it should be possible
> to reproduce most issues on Linux as Linux. Sometimes, it's just that
> some specific Windows buildbot workers are slower than other buildbot
> workers.
>
> Good luck ;-)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGEOKWD3PBWIL2LPZWZ6MMRANN5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7K4Z3QJTZCRALCIIFEJSQXVCMBYU37J2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Awareness creation of a new IDE

2019-10-21 Thread Contact - Google 

Dear Sir / Madam ,

I'm Lasan Nishshanka, CEO and the Founder of Clenontec. We are creating 
an IDE. It's name is Bacend Studio. We're creating this for the Windows 
Operating System. This IDE is designed to perform special functions such 
as App Development, Software Development, Games Development, Machine 
Learning, Artificial Intelligence etc.
We will send you this letter to get permission to add the PYTHON 
Programming Language to our IDE. If you can give it permission, please 
tell us. We await a speedy reply from you.


Company Web Site : https://www.clenontec.com/

E Mail : clenon...@gmail.com

Thank You.


--
This email has been checked for viruses by AVG.
https://www.avg.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LKL5DNJAGAQLNPVJF5DVGIBITZZ2DMBA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Awareness creation of a new IDE

2019-10-21 Thread Guido van Rossum 
Hi Lasan,

You don't need permission to support Python in a commercial IDE. It's open
source. If you use the Python logo, make sure it refers to Python and that
you don't claim to be endorsed by the PSF, that's all we ask.

Good luck!

--Guido

On Mon, Oct 21, 2019 at 9:41 AM Contact - Google 
wrote:

> Dear Sir / Madam ,
>
> I'm Lasan Nishshanka, CEO and the Founder of Clenontec. We are creating
> an IDE. It's name is Bacend Studio. We're creating this for the Windows
> Operating System. This IDE is designed to perform special functions such
> as App Development, Software Development, Games Development, Machine
> Learning, Artificial Intelligence etc.
> We will send you this letter to get permission to add the PYTHON
> Programming Language to our IDE. If you can give it permission, please
> tell us. We await a speedy reply from you.
>
> Company Web Site : https://www.clenontec.com/
>
> E Mail : clenon...@gmail.com
>
> Thank You.
>
>
> --
> This email has been checked for viruses by AVG.
> https://www.avg.com
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LKL5DNJAGAQLNPVJF5DVGIBITZZ2DMBA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OJ5DAQYXZXFUR6NF7BGRFNYQDY2CAW6R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Awareness creation of a new IDE

2019-10-21 Thread Brett Cannon 
If you want to use a modified version of the logo you will need to clear it 
with the PSF trademark WG: 
https://www.python.org/psf/committees/#trademarks-work-group.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FHI5YVIE3QIZXIYZDMMVCPR6TYFPHI4U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Awareness creation of a new IDE

2019-10-21 Thread Kyle Stanley 
> You don't need permission to support Python in a commercial IDE. It's
open source.

Adding to this, see
https://docs.python.org/3/license.html#terms-and-conditions-for-accessing-or-otherwise-using-python
for the complete licensing information on Python.

On a somewhat related note to the licensing page, I just noticed that the
link to Zope's website (http://www.zope.com) is no longer valid. They
currently use a .org TLD instead of .com (https://www.zope.org). Mind if I
open a PR to fix/update the link in Doc/licensing.rst?

On Mon, Oct 21, 2019 at 1:53 PM Guido van Rossum  wrote:

> Hi Lasan,
>
> You don't need permission to support Python in a commercial IDE. It's open
> source. If you use the Python logo, make sure it refers to Python and that
> you don't claim to be endorsed by the PSF, that's all we ask.
>
> Good luck!
>
> --Guido
>
> On Mon, Oct 21, 2019 at 9:41 AM Contact - Google 
> wrote:
>
>> Dear Sir / Madam ,
>>
>> I'm Lasan Nishshanka, CEO and the Founder of Clenontec. We are creating
>> an IDE. It's name is Bacend Studio. We're creating this for the Windows
>> Operating System. This IDE is designed to perform special functions such
>> as App Development, Software Development, Games Development, Machine
>> Learning, Artificial Intelligence etc.
>> We will send you this letter to get permission to add the PYTHON
>> Programming Language to our IDE. If you can give it permission, please
>> tell us. We await a speedy reply from you.
>>
>> Company Web Site : https://www.clenontec.com/
>>
>> E Mail : clenon...@gmail.com
>>
>> Thank You.
>>
>>
>> --
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LKL5DNJAGAQLNPVJF5DVGIBITZZ2DMBA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJ5DAQYXZXFUR6NF7BGRFNYQDY2CAW6R/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RTSYN4K3FD5WX4GANRMUB6QL6YZPAADF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Fun with Python 3.8 and Qt

2019-10-21 Thread Kacvinsky, Tom 
Today I discovered the this struct

typedef struct{
const char* name;
int basicsize;
int itemsize;
unsigned int flags;
PyType_Slot *slots; /* terminated by slot==0. */
} PyType_Spec;

with "PyTypeSlot *slots" being on line 190 of object.h causes a problem when 
compiled with code that brings in Qt.  Qt has macro definitions of slots.

With a cursory preprocessing of the file I was working with, using the handy 
gcc options -dM -E, I found that
slots was defined to nothing

#define slots

and hence caused problems when object.h was brought into the mix.

I will try to make a simple reproducer tomorrow.  I know this probably could be 
solved by header file inclusion re-ordering,
or in some cases #undef'ing slots before including Python.h, but I also thought 
the Python dev team would like to know
about this issue.

Tom
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5VRZVIMFB5OMX7SVQCXCH7FT5MCTN26J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Fun with Python 3.8 and Qt

2019-10-21 Thread Benjamin Peterson 
It's known: https://bugs.python.org/issue38007

On Mon, Oct 21, 2019, at 20:11, Kacvinsky, Tom wrote:
> Today I discovered the this struct
> 
> typedef struct{
> const char* name;
> int basicsize;
> int itemsize;
> unsigned int flags;
> PyType_Slot *slots; /* terminated by slot==0. */
> } PyType_Spec;
> 
> with "PyTypeSlot *slots" being on line 190 of object.h causes a problem 
> when compiled with code that brings in Qt.  Qt has macro definitions of 
> slots.
> 
> With a cursory preprocessing of the file I was working with, using the 
> handy gcc options -dM -E, I found that
> slots was defined to nothing
> 
> #define slots
> 
> and hence caused problems when object.h was brought into the mix.
> 
> I will try to make a simple reproducer tomorrow.  I know this probably 
> could be solved by header file inclusion re-ordering,
> or in some cases #undef'ing slots before including Python.h, but I also 
> thought the Python dev team would like to know
> about this issue.
> 
> Tom
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/5VRZVIMFB5OMX7SVQCXCH7FT5MCTN26J/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QGEQSRONGVVPSNH3ZYYLYQVKECATENJY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Awareness creation of a new IDE

2019-10-21 Thread Terry Reedy 

On 10/21/2019 7:34 PM, Kyle Stanley wrote:

On a somewhat related note to the licensing page, I just noticed that 
the link to Zope's website (http://www.zope.com) is no longer valid. 
They currently use a .org TLD instead of .com (https://www.zope.org). 
Mind if I open a PR to fix/update the link in Doc/licensing.rst?


Please do and ask for my review.  I already verified the fix.

--
Terry Jan Reedy
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XR2GB3OQQVDSHZMUJ7VIEQRC4L65P5NB/
Code of Conduct: http://python.org/psf/codeofconduct/