Re: [Python-Dev] Pycon and Randall Munroe
On Feb 9, 2009, at 2:51 PM, Guido van Rossum wrote: Very cool. Has anyone contacted Randall about giving a keynote at PyCon? Yes, haven't heard back yet. -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Feb 22, 2009, at 5:15 PM, Martin v. Löwis wrote:
What is the objective of this code? Is it a complete sandbox?
If not, is a complete sandbox based on it available somehow for
review?
From a cursory look at Tav's CPython patch, I'd describe it as follows:
Requires: an existing Python code execution environment
which does not expose unsafe interpreter
functionality to the (untrusted) code it runs,
Provides: a way for the trusted code outside of that
restricted environment to wrap functions
(which may reference trusted objects) in
closures which can be passed into the restricted
environment as completely opaque objects
whose internal state (including any referenced
trusted objects) cannot be accessed from the
untrusted code.
When I say 'requires', I mean 'requires to be useful'; the patch can
be applied to vanilla CPython, but isn't useful on its own.
Building blocks for a restricted execution environment as required by
the patch are commonly provided in templating libraries; Tav mentions
Genshi in particular. By default, Genshi templates don't come with
security built in -- you can do what you please in a template.
However, since Genshi manually constructs a Python AST from Python
code in the template, one can restrict the AST and modify the builtins
exposed to the template environment, making things like filesystem
access, import and other sensitive system facilities unavailable.
Even with those unavailable, untrusted code can break out of
containment by accessing object.__subclasses__ or gaining access to
gi_frame and gi_code. This means you can't, for instance, pass into
the untrusted code a closure which operates on trusted objects. Tav's
patch addresses this particular problem.
To give you a complete sandbox to play with, I just violently ripped
out the relevant code from Genshi, added some glue, and slapped it all
together in a single file along with Tav's pure-Python code which is
functionally equivalent to the CPython patch he submitted. The result,
tested on 2.5.1 and nothing else:
<http://radian.org/~krstic/sandbox.py>
As is, sandbox allows you to execute untrusted Python code which won't
have access to import, __import__, eval, exec, execfile, open, and to
which you can pass closures which reference trusted objects which the
untrusted code can't get at.
Example:
>>> import sandbox
>>> import sys
>>> import md5
>>> def trusted_pwd_closure(password):
...def get_pwd():
...return md5.md5(password).hexdigest()
...return get_pwd
...
>>> context = dict(pwd=trusted_pwd_closure('secret'))
>>> sandbox.run_string("print pwd()", context)
>>> sandbox.run_string("print pwd.func_closure[0].cell_contents",
context)
[snip]
AttributeError: 'function' object has no attribute 'func_closure'
Without Tav's patch, the untrusted code can extract our password
('secret') from the closure with that last statement.
To run whole files in the sandbox:
>>> sandbox.run_file('/some/path/file.py')
Finally, disclaimer: I put this together in *15 minutes*. I'm sure I
missed something, this isn't secure, etc. It's just a proof of
concept. Void where prohibited, etc.
Cheers,
--
Ivan Krstić | http://radian.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Feb 22, 2009, at 9:43 PM, Guido van Rossum wrote: I'm not familiar with Genshi -- what is the purpose of the AST transformation here? Sorry, I should have been clearer. If the only goal is to provide a restricted bare interpreter, you can certainly just exec with a restricted set of builtins and no __import__. Since Tav mentioned wanting restricted execution of Genshi templates in particular (which have a rather complicated mechanism for executing inline Python code), I threw together a realistic, self-contained 'restricting Genshi' demo which doesn't rely on outside restrictions, such as those provided by GAE. You can ignore the AST stuff; Genshi does it for its own (non- security) purposes. -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 372 -- Adding an ordered directory to collections ready for pronouncement
On Mar 2, 2009, at 7:08 PM, Steve Holden wrote: PS.: so is datetime.datetime a builtin then? :) Another historic accident. Like socket.socket. :-( A pity this stuff wasn't addressed for 3.0. Way too late now, though. It may be too late to rename the existing accidents, but why not add consistently-named aliases (socket.Socket, datetime.DateTime, etc) and strongly encourage their use in new code? -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa
Glyph, On Mar 4, 2009, at 12:31 AM, gl...@divmod.com wrote: If someone who has PEP-writing skills and some free time is updating asyncore to be more modern, I'd be very happy to walk that person through Twisted's API design and explain how either portions of our implementation could be lifted for the stdlib, or how a "high level" layer could be written into asyncore so that the author of a particular chunk of low-level networking code could ignore whether they're using the stdlib mainloop or one of the Twisted ones. I spent about a half hour sometime in the last month talking this through with Itamar, though not in great detail. I'd be interested in sitting down with both of you and trying to establish more precisely how much work is necessary to get something to actually happen here. I won't outright promise a PEP, but I'll promise at the very least to write down elaborate notes that someone could turn into a PEP relatively straightforwardly. Deal? Cheers, -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Mar 4, 2009, at 12:32 PM, James Y Knight wrote: I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. While we're talking about alternatives, Ian Bicking appears to swear by lxml: <http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ > Cheers, -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IDLE timeout.
On Apr 15, 2009, at 10:05 AM, Alessio Giovanni Baroni wrote: r, w, x = select.select([self.sock.fileno()], [], [], wait) select.error: (4, 'Interrupted system call') See here for an explanation of the same problem in another module: <http://mail.python.org/pipermail/python-dev/2000-October/009671.html> Sounds like you ought to file a bug against IDLE to have it grow EINTR handling. Cheers, -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a token
On Aug 7, 2010, at 9:15 PM, Greg Ewing wrote: > Does anyone know if there's a way to tell Apple's linker to > use a framework from a specific location and not go looking > anywhere else? $ DYLD_FRAMEWORK_PATH= See dyld(1) for other relevant magic. Cheers, -- Ivan Krstić, via mobile > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sad state of OS X Python testing...
On Oct 2, 2010, at 11:50 AM, Bill Janssen wrote: Perhaps we could get Apple to contribute some "seconds"? If you don't get a good solution soon, let me know off-list and I'll see if Apple can help. Cheers, -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Trap SIGSEGV and SIGFPE
Hi Martin, On Dec 11, 2008, at 12:12 AM, Martin v. Löwis wrote: Several people already said (essentially) that: -1. I don't think such code should be added to the Python core, no matter how smart or correct it is. does your -1 apply only to attempts to resume execution after SIGSEGV, or also to the idea of dumping the stack and immediately exiting? The former strikes me as crazy talk, while the latter is genuinely useful. Cheers, -- Ivan Krstić <[EMAIL PROTECTED]> | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Trap SIGSEGV and SIGFPE
On Dec 11, 2008, at 3:05 PM, Martin v. Löwis wrote: If it is actually possible to print a stack trace, that could be useful indeed. I'm then skeptical that this is possible in the general case (i.e. displaying the full C stack), but displaying (parts of) the Python stack might be possible. I think it should still proceed to dump core, so that you can then inspect the core with a proper debugger. +1. Victor, any interest in attempting to retool your patch in this direction? -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
On Dec 13, 2008, at 5:47 PM, Martin v. Löwis wrote: They were originally invented in 1965, on Multics (1970) they were used to perform compilation in the background. When Unix came along, it *added* address space separation, introducing what is now known as processes. Yes, and a lot of the subsequent interest in threads came due to the historically debilitating overhead of fork() on some important Unices, notably Solaris. -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 1:13 PM, Mike Coleman wrote:
On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg wrote:
BTW: Rather than using a huge in-memory dict, I'd suggest to either
use an on-disk dictionary such as the ones found in mxBeeBase or
a database.
I really want this to work in-memory. I have 64G RAM, and I'm only
trying to use 45G of it ("only" 45G :-), and I don't need the results
to persist after the program finishes.
It's still not clear to me, from reading the whole thread, precisely
what you're seeing. A self-contained test case, preferably with
generated random data, would be great, and save everyone a lot of
investigation time. In the meantime, can you 1) turn off all swap
files and partitions, and 2) confirm positively that your CPU cycles
are burning up in userland?
(In general, unless you know exactly why your workload needs swap, and
have written your program to take swapping into account, having _any_
swap on a machine with 64GB RAM is lunacy. The machine will grind to a
complete standstill long before filling up gigabytes of swap.)
--
Ivan Krstić | http://radian.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 4:07 PM, M.-A. Lemburg wrote: What kinds of objects are you storing in your dictionary ? Python instances, strings, integers ? Answered in a previous message: On Dec 20, 2008, at 8:09 PM, Mike Coleman wrote: The dict keys were all uppercase alpha strings of length 7. I don't have access at the moment, but maybe something like 10-100M of them (not sure how redundant the set is). The values are all lists of pairs, where each pair is a (string, int). The pair strings are of length around 30, and drawn from a "small" fixed set of around 60K strings (). As mentioned previously, I think the ints are drawn pretty uniformly from something like range(1). The length of the lists depends on the redundancy of the key set, but I think there are around 100-200M pairs total, for the entire dict. (If you're curious about the application domain, see 'http://greylag.org '.) -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 6:28 PM, Mike Coleman wrote: For (2), yes, 100% CPU usage. 100% _user_ CPU usage? (I'm trying to make sure we're not chasing some particular degeneration of kmalloc/vmalloc and friends.) -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Infra issues (was: Re: I would like an svn account)
On Jan 3, 2009, at 5:47 PM, Martin v. Löwis wrote: There have been many problems on upgrade for the cases where we gave in: shared libraries were missing after the upgrade (for Zope), the software wasn't available anymore after the upgrade (in case of manually-install Python pacakges), and so on. Very few people have actually helped in fixing these problems What's the preferred way of offering help with infrastructure problems, and to what extent, in your opinion, is the solution to have more hands on deck vs. farming out certain (groups of) services to different machines? -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
