Re: [Python-Dev] Unicode literals in Python 2.7
I already have a solution in Python 3 (see https://github.com/Drekin/win-unicode-console, https://pypi.python.org/pypi/win_unicode_console), I was just considering adding support for Python 2 as well. I think I have an working example in Python 2 using ctypes. On Thu, May 7, 2015 at 9:23 PM, "Martin v. Löwis" wrote: > Am 02.05.15 um 21:57 schrieb Adam Bartoš: > > Even if sys.stdin contained a file-like object with proper encoding > > attribute, it wouldn't work since sys.stdin has to be instance of > 'file'>. So the question is, whether it is possible to make a file > instance > > in Python that is also customizable so it may call my code. For the first > > thing, how to change the value of encoding attribute of a file object. > > If, by "in Python", you mean both "in pure Python", and "in Python 2", > then the answer is no. If you can add arbitrary C code, then you might > be able to hack your C library's stdio implementation to delegate fread > calls to your code. > > I recommend to use Python 3 instead. > > Regards, > Martin > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode literals in Python 2.7
On 5/9/2015 5:39 AM, Adam Bartoš wrote: I already have a solution in Python 3 (see https://github.com/Drekin/win-unicode-console, https://pypi.python.org/pypi/win_unicode_console), I was just considering adding support for Python 2 as well. I think I have an working example in Python 2 using ctypes. Is this going to get released in 3.5, I hope? Python 3 is pretty limited without some solution for Unicode on the console... probably the biggest deficiency I have found in Python 3, since its introduction. It has great Unicode support for files and processing, which convinced me to switch from Perl, and I like so much else about it, that I can hardly code in Perl any more (I still support a few Perl programs, but have ported most of them to Python). I wondered if all your recent questions about Py 2 were as a result of porting the above to Py 2... I only have one program left that I was forced to write in Py 2 because of library dependencies, and I think that library is finally being ported to Py 3, whew! So while I laud your efforts, and no doubt they will benefit some folks for a few years yet, I hope never to use your Py 2 port myself! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 09.05.2015 02:29, Nick Coghlan wrote: > On 8 May 2015 8:14 pm, "M.-A. Lemburg" wrote: >> >> On 08.05.2015 11:36, Nick Coghlan wrote: >>> On 8 May 2015 6:52 pm, "M.-A. Lemburg" wrote: On 07.05.2015 04:30, Nick Coghlan wrote: >> Can we please make the monkeypatch a regular part of Python's >> site.py which can enabled via an environment variable, say >> export PYTHONHTTPSVERIFY=0. >> >> See http://bugs.python.org/issue23857 for the discussion. > ... > I actually do think it would be good to have such a feature as a > native part of Python 2.7 in order to provide a nicer "revert to the > pre-PEP-476 behaviour" experience for Python 2.7 users (leaving the > "there's no easy way to turn HTTPS certificate verification off > globally" state of affairs to Python 3), but I don't currently have > the time available to push for that against the "end users can't be > trusted not to turn certificate verification off when they should be > fixing their certificate management instead" perspective. We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. >>> >>> That's a slightly misleading quotation of my post, as I'm opposed to the >>> use of an environment variable for this, due to the fact that using the >>> "-E" switch will then revert to the upstream default behaviour of > verifying >>> certificates, rather defeating the point of introducing the legacy >>> infrastructure compatibility feature in the first place. >> >> Oh, sorry. I read your email implying that you are fine with >> the env var approach. >> >> I don't really see the issue with -E, though. It's well possible >> to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain >> backwards compatibility per default for Python 2.7. >> >> Regarding the config file approach and letting distributions >> set their own defaults: >> >> I don't think it's a good idea to have one distribution >> default to verifying HTTPS certs via a global config file >> and another distribution do the opposite. >> >> Python itself should define the defaults to be used, not >> the distributions. > > As a result of the discussions around both PEP 466 and 476, I'm now firmly > of the view that it's correct for the upstream Python community to assume > the public internet as its standard operating environment when it comes to > network security settings, and for those of us working for commercial > redistributors to subsequently bear the cost of adapting from that upstream > assumption to the different assumptions that may apply in the context of > organisational intranets. > > That's also why I've come around to the view that an informational PEP with > recommendations for redistributors, rather than an actual change to Python > 2.7, is the right answer (at least initially) when it comes to providing a > smoother transition plan for PEP 476 - the folks saying "it's a bad idea to > make this easy to turn off" are *right* from the perspective of operating > over the public internet, or with well designed internal SSL/TLS > certificate management, it's just also a *necessary* idea (in my view) to > help CIOs and other infrastructure leads responsibly and effectively manage > the wide range of risks associated with internal infrastructure upgrades. I don't agree. We've broken the contract that people had with Python 2.7 by introducing a major breakage in a patch level release very far down the line in 2.7.9, without providing an easy and official way to opt-out that does not involve hacking your installation. IMO, we should not fall for the somewhat arrogant view that we know better than all the Python users out there when it comes to running secure systems. By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.
[Python-Dev] Free lists
Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux). typecount % acc.% builtin_function_or_method 116012007 36.29% 36.29% method 52465386 16.41% 52.70% int 42828741 13.40% 66.09% str 37017098 11.58% 77.67% generator14026583 4.39% 82.06% list_iterator 8731329 2.73% 84.79% bytes 7217934 2.26% 87.04% tuple_iterator5042563 1.58% 88.62% float 4672980 1.46% 90.08% set 3319699 1.04% 91.12% _io.StringIO 3000369 0.94% 92.06% str_iterator 2126838 0.67% 92.73% list 2031059 0.64% 93.36% dict 1691993 0.53% 93.89% method-wrapper1573139 0.49% 94.38% function 1472062 0.46% 94.84% traceback 1388278 0.43% 95.28% tuple 1132071 0.35% 95.63% memoryview1092173 0.34% 95.97% cell 1049496 0.33% 96.30% managedbuffer 1036889 0.32% 96.63% bytearray 711969 0.22% 96.85% range_iterator 496924 0.16% 97.00% range 483971 0.15% 97.15% super 472447 0.15% 97.30% map449567 0.14% 97.44% frame 427320 0.13% 97.58% set_iterator 423392 0.13% 97.71% Leaf 398705 0.12% 97.83% symtable 374412 0.12% 97.95% Types for which free lists already are used: builtin_function_or_method, method, float, tuple, list, dict, frame. Some free list implementations (e.g. for tuple) don't call PyObject_INIT/PyObject_INIT_VAR. That is why numbers are such low for tuples. Perhaps it is worth to add free lists for other types: int, str, bytes, generator, list and tuple iterators? Shortened tables for variable-sized objects (that calls PyObject_INIT_VAR): int 42828741 13.40% 0 425353 0.99% 0.99% 1 21399290 49.96% 50.96% 2 10496856 24.51% 75.47% 34873346 11.38% 86.85% 41021563 2.39% 89.23% 51246444 2.91% 92.14% 6 733676 1.71% 93.85% 7 123074 0.29% 94.14% 8 139203 0.33% 94.47% ... bytes 7217934 2.26% 0842 0.01% 0.01% 1 179469 2.49% 2.50% 2 473306 6.56% 9.06% 3 254968 3.53% 12.59% 41169164 16.20% 28.79% 5 72806 1.01% 29.79% 6 128668 1.78% 31.58% 7 169694 2.35% 33.93% 8 155154 2.15% 36.08% 9 67320 0.93% 37.01% 10 51703 0.72% 37.73% 11 42574 0.59% 38.32% 12 108947 1.51% 39.83% 13 40812 0.57% 40.39% 14 126783 1.76% 42.15% 15 37873 0.52% 42.67% 16 447482 6.20% 48.87% 17 194320 2.69% 51.56% 18 251685 3.49% 55.05% 19 159435 2.21% 57.26% 20 212521 2.94% 60.20% ... 31 18751 0.26% 67.32% 32 159781 2.21% 69.54% 33 8332 0.12% 69.65% ... 63 19841 0.27% 79.21% 64 144982 2.01% 81.22% 65 5216 0.07% 81.29% ... 127 1354 0.02% 85.44% 128 376539 5.22% 90.66% 129 17468 0.24% 90.90% ... 255178 0.00% 92.39% 256 11993 0.17% 92.55% 257124 0.00% 92.56% ... ___ Pyt
Re: [Python-Dev] Free lists
On 05/09/2015 12:01 PM, Serhiy Storchaka wrote: Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux). Can you produce these statistics for a 64-bit build? //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Free lists
On 09.05.15 22:51, Larry Hastings wrote: On 05/09/2015 12:01 PM, Serhiy Storchaka wrote: Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux). Can you produce these statistics for a 64-bit build? Sorry, no. All my computers are ran under 32-bit Linux. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Free lists
On May 9, 2015 5:07 PM, "Serhiy Storchaka" wrote: > > On 09.05.15 22:51, Larry Hastings wrote: >> >> On 05/09/2015 12:01 PM, Serhiy Storchaka wrote: >>> >>> Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR >>> for types (collected during running Python tests on 32-bit Linux). >> >> >> Can you produce these statistics for a 64-bit build? > > > Sorry, no. All my computers are ran under 32-bit Linux. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com Can you share how you gathered them so someone could run them on a 64-bit build? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg wrote: > By providing a way to intentionally switch off the new default, > we do make people aware of the risks and that's good enough, > while still maintaining the contract people rightly expect of > patch level releases of Python. Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10 May 2015 at 11:44, Chris Angelico wrote: > On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg wrote: >> By providing a way to intentionally switch off the new default, >> we do make people aware of the risks and that's good enough, >> while still maintaining the contract people rightly expect of >> patch level releases of Python. > > Just as long as it's the sysadmin, and NOT some random attacker over > the internet, who has the power to downgrade security. Environment > variables can be attacked in various ways. They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PYTHONHTTPSVERIFY env var
On 10 May 2015 at 13:04, Robert Collins wrote: > On 10 May 2015 at 11:44, Chris Angelico wrote: >> On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg wrote: >>> By providing a way to intentionally switch off the new default, >>> we do make people aware of the risks and that's good enough, >>> while still maintaining the contract people rightly expect of >>> patch level releases of Python. >> >> Just as long as it's the sysadmin, and NOT some random attacker over >> the internet, who has the power to downgrade security. Environment >> variables can be attacked in various ways. > > They can, and the bash fun was very good evidence of that. > > OTOH if someones environment is at risk, PATH and PYTHONPATH are > already very effective attack vectors. Right, which is why -E is an important existing hardening technique for protecting privileged services against local attackers. Isolated mode in Python 3.4+ is easier to use, but you can get a functional equivalent in Python 2 using: * running from a directory under /usr (Program Files on Windows) rather than your home directory (to protect against sys.path[0] based attacks) * running with -E (to protect against PYTHON* environment variable attacks) * running with -S (to protect against site.py and sitecustomize.py based attacks) * running with -s (to protect against hostile packages in the user site directory) That's how I came to the conclusion that adding a new environment variable to turn off a network security hardening feature isn't a good idea: * it significantly increases the attack surface area if you're *not* using -E when running a privileged service * it doesn't work at all if you *are* using -E when running a privileged service That was OK when we were dealing with the hash randomisation problem, mostly because the consequence of that vulnerability was "denial of service", and the question of whether or not hash randomisation caused problems came up on an application-by-application basis, rather than being related to the way an entire network environment was managed. The question becomes very different when the failure mode we're talking about is transparent interception of nominally confidential communication. Instead, we want a configuration file stored in a protected directory, such that for an attacker to modify it they *already* need to have achieved a local privilege escalation, in which case, they can just attack the system certificate store directly, rather than messing about downgrading Python's default HTTPS verification settings. In my case, I don't actually need the *feature itself* in upstream CPython, but I *would* like to have upstream CPython's blessing of the design as a recommendation to redistributors that need a capability like this to meet the needs of their end users. I've been talking about "someone" putting together a PEP to that effect, so given this discussion, I'll go ahead and do that myself, with Robert Kuska listed as co-author (since he came up with the general design I'm advocating for). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Free lists
On 10.05.15 02:25, Ian Cordasco wrote:
Can you share how you gathered them so someone could run them on a
64-bit build?
This is quick and dirty patch. It generates 8 GB log file!
patch --merge -p1 PyObject_INIT.log
python3 PyObject_INIT_stat.py PyObject_INIT.stat
Perhaps compiling with COUNT_ALLOCS will produce similar statistic for
types (but without statistics for sizes) and should be much faster.
diff -r f7cc54086cd2 Include/objimpl.h
--- a/Include/objimpl.h Sat May 09 11:37:23 2015 -0700
+++ b/Include/objimpl.h Sat May 09 22:05:04 2015 +0300
@@ -137,9 +137,9 @@ PyAPI_FUNC(PyVarObject *) _PyObject_NewV
/* Macros trading binary compatibility for speed. See also pymem.h.
Note that these macros expect non-NULL object pointers.*/
#define PyObject_INIT(op, typeobj) \
-( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )
+( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), fprintf(stderr, "PyObject_INIT %.200s\n", Py_TYPE(op)->tp_name), (op) )
#define PyObject_INIT_VAR(op, typeobj, size) \
-( Py_SIZE(op) = (size), PyObject_INIT((op), (typeobj)) )
+( Py_SIZE(op) = (size), Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), fprintf(stderr, "PyObject_INIT_VAR %.200s %zd\n", Py_TYPE(op)->tp_name, Py_SIZE(op)), (op) )
#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )
import sys, collections
stat1 = collections.Counter()
stat2 = collections.defaultdict(collections.Counter)
for line in sys.stdin:
if not line.startswith('PyObject_INIT'):
continue
try:
s, t, *r = line.split()
stat1[t] += 1
if s == 'PyObject_INIT_VAR':
stat2[t][int(r[0])] += 1
except Exception as e:
print('*** ERROR: %r %r' % (line, e), file=sys.stderr)
total = sum(stat1.values())
acc = 0
print('%-30s %10s %7s %7s' % ('type', 'count', '%', 'acc.%'))
print()
for t, c in stat1.most_common(30):
acc += c
print('%-30s %10d %6.2f%% %6.2f%%' % (t, c, 100 * c / total, 100 * acc / total))
for t in sorted(stat2, key=stat1.__getitem__, reverse=True)[:20]:
c = stat1[t]
print()
print('%-30s %10d %6.2f%%' % (t, c, 100 * c / total))
acc = 0
for s, c2 in sorted(stat2[t].items()):
acc += c2
print('%30d %10d %6.2f%% %6.2f%%' % (s, c2, 100 * c2 / c, 100 * acc / c))
c2 = c - sum(stat2[t].values())
if c2:
print('%30s %10d %6.2f%% %6.2f%%' % ('-', c2, 100 * c2 / c, 100))
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
