Re: [Python-Dev] Setting project home path the best way

[Christian Tismer]

Hi Daniel,

no, I was not aware of this. I just read it up on

http://sayspy.blogspot.de/2010/03/various-ways-of-distributing-python.html

Yeah, thank you very much for this hint, very useful! ;-)

cheers - Chris

On 16.11.12 04:22, Daniel Holth wrote:

Are you familiar with executing directories having __main__.py as python 
scripts?

Daniel Holth

On Nov 15, 2012, at 4:43 PM, Christian Tismer  wrote:


Hi Kristjan,

does that mean that your scheme simply works, without any config step
necessary after I did my checkout?
This would in fact be an interesting alternative to

Python setup.py develop

but I'm not sure if this is the same scheme on windows and Os X.

Getting this part right was rather tricky, and I fear this is still an issue.

Right now I think to just force my users to run the install step, since it is 
quite
accepted in general.

Still, I'd love to see a way with no action needed at all: write yout your 
structure,
and it works as-is. Seems to be impossible without tricks.

Cheers - chris

Sent from my Ei4Steve

On Nov 15, 2012, at 10:17, Kristján Valur Jónsson  wrote:


When python is being run from a compile environment, it detects this by looking for 
"Lib" folders in directories above the one containing the executable.
(I always thought that this "special" execution mode, hardwired in, was a bit 
odd, and suggested that this could be made a function of pep405)
Anyway, keeping your executable as part of the tree is the trick I use, and to 
make things nice I put  right next to it:
site.py
sitecustomize.py

sitecustomize.py is where you would put the logic to set sys.path by walking up 
the hierarchy and finding the proper root.
site.py is there to merely import sitecustomize.py, in case a site.py is not 
found in all the default places python looks.

K



-Original Message-
From: Python-Dev [mailto:python-dev-
bounces+kristjan=ccpgames@python.org] On Behalf Of Christian Tismer
Sent: 11. nóvember 2012 20:31
To: python-dev@python.org
Subject: [Python-Dev] Setting project home path the best way

Hi friends,

I have a project that has its root somewhere on my machine.
This project has many folders and contains quite some modules.

There is a common root of the module tree, and I want to use
- either absolute imports
- relative imports with '.'

Problem:

- I want to run any module inside the heirarchy from the command-line

- this should work, regardless what my 'cwd' is

- this should work with or without virtualenv.

So far, things work fine with virtualenv, because sys.executable is in the
project module tree.

Without virtualenv, this is not so. But I hate to make settings like
PYTHONPATH, because these are not permanent. .

Question:

How should I define my project root dir in a unique way, without setting an
environment variable?
What is the lest intrusive way to spell that?

Reason:

I'd like to make things work correctly and unambigously when I call a script
inside the module heirarchy. Things are not fixed: there exist many
checkouts In the file system, and each should know where to search its
home/root in the tree.

Is this elegantly possible to deduce from the actually executed script file?

Cheers - chris

Sent from my Ei4Steve
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-
dev/kristjan%40ccpgames.com



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com



--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] externals?

[Trent Nelson]

On Thu, Nov 15, 2012 at 01:20:09AM -0800, Kristj?n Valur J?nsson wrote:
> Perhaps the unix makefiles get the proper version, but a windows developer 
> has to fetch those externals manually.

Pro-tip: if you're developing on Windows, you're mad if you don't
prime your dev env with Tools\buildbot\externals.bat.  It takes
care of *everything*.  I wish every proprietary UNIX system we
support had something similar.

> Also, is there any reason to keep this in svn?

I think it's more a case of there being no tangible benefit (and
numerous drawbacks) to switching it to hg.  I personally have no
need for a local hg repo with 30 different Tcl/Tk histories in
it.

Subversion's good for this sort of use case.  The externals repo
gets committed to maybe, what, 3-4 times a year?

> Why not check this in to HG, we need not worry about history, etc.

Who are these mystical people worried about history? ;-)

Trent.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generally bored by installation

[Eli Bendersky]

>Christian, I feel your pain. Everyone does, I am sure. Who among us
> never cursed that dratted computers? Those dirty rotten things that
> always hang due to bugs in drivers or whatnot, require constant
> cleaning, mending, upgrading, repairing, debugging. Condemn that magical
> mouse gestures, be cursed those magical incantations. Who in his normal
> mind would ever think of writing things like
>
> exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name
> .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o
> -type f -exec grep -I "$@" '{}' \+ 2>/dev/null
>
>???!!!
>
>
Use pss ;-) [http://pypi.python.org/pypi/pss]

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pss (was: Generally bored by installation)

[Oleg Broytman]

On Fri, Nov 16, 2012 at 05:34:37AM -0800, Eli Bendersky  
wrote:
> > exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name
> > .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o
> > -type f -exec grep -I "$@" '{}' \+ 2>/dev/null
> >
> Use pss ;-) [http://pypi.python.org/pypi/pss]

   Hardly. I am proficient in find+grep. And pss is underdocumented.

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2012-11-09 - 2012-11-16)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open3801 ( -5)
  closed 24464 (+46)
  total  28265 (+41)

Open issues with patches: 1660 


Issues opened (27)
==

#13863: import.c sometimes generates incorrect timestamps on Windows +
http://bugs.python.org/issue13863  reopened by eric.snow

#16450: test_missing_localfile masks problems in urlopen
http://bugs.python.org/issue16450  opened by HansM

#16451: Remove duplication between slice_indices and compute_slice_ind
http://bugs.python.org/issue16451  opened by mark.dickinson

#16455: sys.getfilesystemencoding() is not the locale encoding on Free
http://bugs.python.org/issue16455  opened by haypo

#16458: subprocess.py throw "The handle is invalid" error on duplicati
http://bugs.python.org/issue16458  opened by kartlee05

#16461: wave module: wrong integer format
http://bugs.python.org/issue16461  opened by ckern

#16462: smtpd should return greeting
http://bugs.python.org/issue16462  opened by mike.a

#16463: test_timeout failure on the RHEL buildbot
http://bugs.python.org/issue16463  opened by pitrou

#16464: urllib.request: opener not resetting content-length
http://bugs.python.org/issue16464  opened by terry.reedy

#16465: dict creation performance regression
http://bugs.python.org/issue16465  opened by phihag

#16466: register command forgets password if no config file is created
http://bugs.python.org/issue16466  opened by techtonik

#16470: Backport set and dictionary comprehensions in tutorial to 2.7
http://bugs.python.org/issue16470  opened by fossilet

#16471: upgrade to sphinx 1.1
http://bugs.python.org/issue16471  opened by chris.jerdonek

#16472: Distutils+mingw links agains msvcr90, while python27.dll is li
http://bugs.python.org/issue16472  opened by eudoxos

#16473: Minor difference in decoding quoted-printable text
http://bugs.python.org/issue16473  opened by aleperalta

#16474: More code coverage for imp module
http://bugs.python.org/issue16474  opened by tenuki

#16475: Support object instancing and recursion in marshal
http://bugs.python.org/issue16475  opened by kristjan.jonsson

#16477: tarfile fails to close file handles in case of exception
http://bugs.python.org/issue16477  opened by ssam

#16480: pyvenv 3.3 fails to create symlinks for /local/{bi
http://bugs.python.org/issue16480  opened by Marco.Amadori

#16482: pdb.set_trace() clobbering traceback on error
http://bugs.python.org/issue16482  opened by akaptur

#16483: Make int(float('inf')) raise ValueError rather than OverflowEr
http://bugs.python.org/issue16483  opened by mark.dickinson

#16484: Missing/broken documentation redirect for http://docs.python.o
http://bugs.python.org/issue16484  opened by mgedmin

#16485: FD leaks in aifc module
http://bugs.python.org/issue16485  opened by serhiy.storchaka

#16486: Add context manager support to aifc module
http://bugs.python.org/issue16486  opened by serhiy.storchaka

#16487: Allow ssl certificates to be speficfied from memory rather tha
http://bugs.python.org/issue16487  opened by kristjan.jonsson

#16488: Add context manager support to epoll object
http://bugs.python.org/issue16488  opened by serhiy.storchaka

#16468: argparse only supports iterable choices
http://bugs.python.org/issue16468  opened by chris.jerdonek



Most recent 15 issues with no replies (15)
==

#16488: Add context manager support to epoll object
http://bugs.python.org/issue16488

#16486: Add context manager support to aifc module
http://bugs.python.org/issue16486

#16484: Missing/broken documentation redirect for http://docs.python.o
http://bugs.python.org/issue16484

#16482: pdb.set_trace() clobbering traceback on error
http://bugs.python.org/issue16482

#16480: pyvenv 3.3 fails to create symlinks for /local/{bi
http://bugs.python.org/issue16480

#16472: Distutils+mingw links agains msvcr90, while python27.dll is li
http://bugs.python.org/issue16472

#16471: upgrade to sphinx 1.1
http://bugs.python.org/issue16471

#16464: urllib.request: opener not resetting content-length
http://bugs.python.org/issue16464

#16463: test_timeout failure on the RHEL buildbot
http://bugs.python.org/issue16463

#16450: test_missing_localfile masks problems in urlopen
http://bugs.python.org/issue16450

#16446: pdb raises BdbQuit on 'quit' when started with set_trace
http://bugs.python.org/issue16446

#16429: Emit SyntaxWarning for code that risks UnboundLocalError
http://bugs.python.org/issue16429

#16428: turtle with compound shape doesn't get clicks
http://bugs.python.org/issue16428

#16425: minidom replaceChild(new_child, old_child) removes new_child e
http://bugs.python.org/issue16425

#16406: move the "Uploading Packages" section to distutils/packageinde
http://bugs.python.org/issue16406



Most recent 15 issues waiting for review (15)

Re: [Python-Dev] Generally bored by installation

[Vinay Sajip]

Eli Bendersky  gmail.com> writes:

> Use pss  [http://pypi.python.org/pypi/pss]

How does it compare with grin? [http://pypi.python.org/pypi/grin]

On the face of it, they both do the same job.

Regards,

Vinay Sajip



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[Chris Jerdonek]

I filed an issue in the meta tracker about e-mails like the below that
are sent to python-dev.  The issue link is here:

http://psf.upfronthosting.co.za/roundup/meta/issue492

--Chris


On Thu, Nov 15, 2012 at 6:39 PM, Python tracker
 wrote:
>
>
> An unexpected error occurred during the processing
> of your message. The tracker administrator is being
> notified.
>
> Return-Path: 
> X-Original-To: rep...@bugs.python.org
> Delivered-To: roundup+trac...@psf.upfronthosting.co.za
> Received: from mail.python.org (mail.python.org [IPv6:2001:888:2000:d::a6])
> by psf.upfronthosting.co.za (Postfix) with ESMTPS id DFD211C98E
> for ; Fri, 16 Nov 2012 03:39:49 +0100 (CET)
> Received: from albatross.python.org (localhost [127.0.0.1])
> by mail.python.org (Postfix) with ESMTP id 3Y2kDj4Tk8zRb6
> for ; Fri, 16 Nov 2012 03:39:49 +0100 (CET)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901;
> t=1353033589; bh=H41Haza/UPvj9B16qvXQtqjlb22jRmazur238MqAPOE=;
> h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To:
>  Subject:Message-Id:Date;
> b=yz1YIEtIYuM3H9V1Ok0YJ/SDvU8xtO+cFOApgDH8jPRdxRS6z/CSdZ96bSY1cNFM7
>  4xxKY3LiUXipe439DPwEjxZr0HcsfX+2JdR4Pzf3OZmdopV6PEl8/twaIFoTHQMy8u
>  dTwpZ7G4OJKCc2IeCq4e5Bl/TEvQvVT9NO8eoa3k=
> Received: from localhost (HELO mail.python.org) (127.0.0.1)
>   by albatross.python.org with SMTP; 16 Nov 2012 03:39:49 +0100
> Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org 
> [140.211.10.72])
> by mail.python.org (Postfix) with ESMTP
> for ; Fri, 16 Nov 2012 03:39:49 +0100 (CET)
> MIME-Version: 1.0
> Content-Type: text/plain; charset="utf-8"
> Content-Transfer-Encoding: base64
> From: python-dev@python.org
> To: rep...@bugs.python.org
> Subject: [issue15894]
> Message-Id: <3y2kdj4tk8z...@mail.python.org>
> Date: Fri, 16 Nov 2012 03:39:49 +0100 (CET)
>
> TmV3IGNoYW5nZXNldCBiZDg1MzMxMWZmZTAgYnkgQnJldHQgQ2Fubm9uIGluIGJyYW5jaCAnZGVm
> YXVsdCc6Cklzc3VlICMxNTg5NDogRG9jdW1lbnQgd2h5IHdlIGRvbid0IHdvcnJ5IGFib3V0IHJl
> LWFjcXVpcmluZyB0aGUKaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvYmQ4NTMzMTFm
> ZmUwCg==
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generally bored by installation

[Eli Bendersky]

On Fri, Nov 16, 2012 at 9:18 AM, Vinay Sajip wrote:

> Eli Bendersky  gmail.com> writes:
>
> > Use pss  [http://pypi.python.org/pypi/pss]
>
> How does it compare with grin? [http://pypi.python.org/pypi/grin]
>
> On the face of it, they both do the same job.
>

They're quite similar. pss is more close to ack, in the sense that it knows
how to identify files belonging to various categories/languages and lets
you easily guide the search this way. And some others bells and whistles.
But generally, both are better than using the horrible "find | xargs grep"
incantation and all its variations.

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pss (was: Generally bored by installation)

[Eli Bendersky]

On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman  wrote:

> On Fri, Nov 16, 2012 at 05:34:37AM -0800, Eli Bendersky 
> wrote:
> > > exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name
> > > .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o
> > > -type f -exec grep -I "$@" '{}' \+ 2>/dev/null
> > >
> > Use pss ;-) [http://pypi.python.org/pypi/pss]
>
>Hardly. I am proficient in find+grep. And pss is underdocumented.
>
>
Even reading https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined
with the built-in help? If anything is missing, by all means let
me know and I'll gladly add it.

[Sorry for the offtopic, folks, promise it's short]

Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generally bored by installation

[Oleg Broytman]

On Fri, Nov 16, 2012 at 01:15:26PM -0800, Eli Bendersky  
wrote:
> But generally, both are better than using the horrible "find | xargs grep"
> incantation and all its variations.

   In what way they are better? I found 'find' to be quite capable, and
find -exec grep '{}' \+
   to be quite fast.

   In my opinion all those small utilities are pets of their authors and
they solve their problems quite good... but not mine. GNU utilities are
used by a huge number of people and solve much wider problems much
better. Though not necessary in a much more elegant way ;-)

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pss

[Oleg Broytman]

On Fri, Nov 16, 2012 at 01:20:17PM -0800, Eli Bendersky  
wrote:
> On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman  wrote:
> >And pss is underdocumented.
> >
> Even reading 
> https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined

   Compare your page with

man find

   and

man grep

   Compare amount of text, levels of details, number of options,
examples...

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pss

[Guido van Rossum]

Oleg, this is inappropriate for python-dev. It's also a bit
uncivilized and unkind.

On Fri, Nov 16, 2012 at 1:40 PM, Oleg Broytman  wrote:
> On Fri, Nov 16, 2012 at 01:20:17PM -0800, Eli Bendersky  
> wrote:
>> On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman  wrote:
>> >And pss is underdocumented.
>> >
>> Even reading 
>> https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined
>
>Compare your page with
>
> man find
>
>and
>
> man grep
>
>Compare amount of text, levels of details, number of options,
> examples...
>
> Oleg.
> --
>  Oleg Broytmanhttp://phdru.name/p...@phdru.name
>Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[Antoine Pitrou]

On Fri, 16 Nov 2012 12:47:52 -0800
Chris Jerdonek  wrote:
> I filed an issue in the meta tracker about e-mails like the below that
> are sent to python-dev.  The issue link is here:
> 
> http://psf.upfronthosting.co.za/roundup/meta/issue492

These e-mails are just because someone mentioned the wrong issue number
in a commit message, I think.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[Guido van Rossum]

But python-dev seems a poor place to spam with those errors.

On Fri, Nov 16, 2012 at 2:41 PM, Antoine Pitrou  wrote:
> On Fri, 16 Nov 2012 12:47:52 -0800
> Chris Jerdonek  wrote:
>> I filed an issue in the meta tracker about e-mails like the below that
>> are sent to python-dev.  The issue link is here:
>>
>> http://psf.upfronthosting.co.za/roundup/meta/issue492
>
> These e-mails are just because someone mentioned the wrong issue number
> in a commit message, I think.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[martin]

Zitat von Guido van Rossum :


But python-dev seems a poor place to spam with those errors.


It's certainly not deliberate that they are sent. However, there
are too few people interested in working on fixing that so that
it remains unfixed. Even finding out why they are sent to python-dev
is tricky.

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[Antoine Pitrou]

On Sat, 17 Nov 2012 00:47:33 +0100
mar...@v.loewis.de wrote:
> 
> Zitat von Guido van Rossum :
> 
> > But python-dev seems a poor place to spam with those errors.
> 
> It's certainly not deliberate that they are sent. However, there
> are too few people interested in working on fixing that so that
> it remains unfixed. Even finding out why they are sent to python-dev
> is tricky.

I think it's because I configured the dummy "python-dev" user that way:
http://bugs.python.org/user13902

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pss

[Oleg Broytman]

On Fri, Nov 16, 2012 at 02:35:13PM -0800, Guido van Rossum  
wrote:
> Oleg, this is inappropriate for python-dev. It's also a bit
> uncivilized and unkind.

   Shame on me! Please, everyone, take my sincerest apology! I'm
stopping now.

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Register-based VM for CPython

[Victor Stinner]

Hi,

I'm still looking for the best approach to speedup CPython. I found the
following article comparing a stack-based VM to a register-based VM which
announces a speed up between 20% and 40% (depending if the instruction
dispatch uses a dummy switch or computed goto):

http://static.usenix.org/events/vee05/full_papers/p153-yunhe.pdf
Yunhe Shi, David Gregg, Andrew Beatty, M. Anton Ertl, 2005

I tried to implement such register-based instructions for CPython in the
following fork of Python 3.4:
http://hg.python.org/sandbox/registervm/

Contact me if you are interested and/or if you would like to contribute!

--

My implementation is not finished and still experimental. It is only
enabled explictly by calling registervm.optimize_func(func) where
registervm is a new module. This function rewrites the bytecode of the
function inplace. I chose this approach because it was simple to implement,
it is simple to compare stack-based and register-based instructions, and it
is also the approach used in the paper :-)

The major drawback of the register approach (at least of my implementation)
is that it changes the lifetime of objects. Newly created objects are only
"destroyed" at the exit of the function, whereas the stack-based VM
destroys "immediatly" objects (thanks to the reference counter). PyPy has
similar issues with its different garbage collector.

On Linux (with computed goto in ceval.c), there are some interesting
speedups in pybench, up to 25% faster. Such speedup can be explained with
the bytecode. For example, "c = a + b" is compiled to:

   LOAD_FAST'a'
   LOAD_FAST'b'
   BINARY_ADD
   STORE_FAST   'c'

The optimized bytecode becomes:

  BINARY_ADD_REG   'c', 'a', 'b'

So 4 operations are replaced with only 1 operation. So instruction
dispatching has a cost, measurable in this microbenchmark.

My implementation is incomplete: if you see "PUSH_REG" or "POP_REG" in the
assembler, your code will be slower than the original bytecode. There is no
register allocator and the optimizer doesn't understand the control flow,
so I expect more performance improvment with a more complete
implementation. Some optimizations are also disabled by default because the
implementation is buggy.

The main changes of registervm are done in Python/ceval.c (implement new
instructions) and Lib/registervm.py (the new module rewriting the
bytecode). Objects/frameobject.c is also modified to allocate registers.

See registervm documentation for more information:
http://hg.python.org/sandbox/registervm/file/tip/REGISTERVM.txt

--

The WPython project is similar to my work (except that it does not use
registers). It tries also to reduce the overhead of instruction dispatch by
using more complex instructions.
http://code.google.com/p/wpython/

Using registers instead of a stack allow to implement more optimizations
(than WPython). For example, it's possible to load constants outside loops
and merge "duplicate constant loads".

I also implemented more aggressive and experimental optimizations (disabled
by default) which may break applications: move loads of attributes and
globals outside of loops, and replace binary operations with inplace
operations. For example, "x=[]; for ...: x.append(...)" is optimized to
something like "x=[]; x_append=x.append; for ...: x_append(...)", and "x =
x + 1" is replaced with "x += 1".

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[R. David Murray]

On Sat, 17 Nov 2012 01:01:52 +0100, Antoine Pitrou  wrote:
> On Sat, 17 Nov 2012 00:47:33 +0100
> mar...@v.loewis.de wrote:
> > 
> > Zitat von Guido van Rossum :
> > 
> > > But python-dev seems a poor place to spam with those errors.
> > 
> > It's certainly not deliberate that they are sent. However, there
> > are too few people interested in working on fixing that so that
> > it remains unfixed. Even finding out why they are sent to python-dev
> > is tricky.
> 
> I think it's because I configured the dummy "python-dev" user that way:
> http://bugs.python.org/user13902

I'm pretty sure it's because python-dev is the 'from' address
used when the messages are sent...and the configuration of
that user is what allows them to be accepted.

I suggest changing the from address and the account configuration
to tracker-disc...@python.org instead.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Failed issue tracker submission

[Chris Jerdonek]

On Fri, Nov 16, 2012 at 10:06 PM, R. David Murray  wrote:
> On Sat, 17 Nov 2012 01:01:52 +0100, Antoine Pitrou  
> wrote:
>> On Sat, 17 Nov 2012 00:47:33 +0100
>> mar...@v.loewis.de wrote:
>> >
>> > Zitat von Guido van Rossum :
>> >
>> > > But python-dev seems a poor place to spam with those errors.
>> >
>> > It's certainly not deliberate that they are sent. However, there
>> > are too few people interested in working on fixing that so that
>> > it remains unfixed. Even finding out why they are sent to python-dev
>> > is tricky.
>>
>> I think it's because I configured the dummy "python-dev" user that way:
>> http://bugs.python.org/user13902
>
> I'm pretty sure it's because python-dev is the 'from' address
> used when the messages are sent...and the configuration of
> that user is what allows them to be accepted.
>
> I suggest changing the from address and the account configuration
> to tracker-disc...@python.org instead.

Above and on the meta tracker issue I filed, it sounds like Martin is
saying that the e-mails should never be going to python-dev (or
tracker-discuss) in the first place -- due to a roundup bug and
because the e-mail is already being sent to the submitter and the
roundup admins.  Thus, changing the from address would only mitigate
the problem and not be fixing the root cause.

--Chris
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com